Devices and Methods for Measuring Using Augmented Reality

ABSTRACT

An electronic device: displays a field of view of a camera at a first magnification and updates the displayed field of view over time based on changes detected by the camera. The field of view includes a view of a three-dimensional space. In response to a first touch input, the device adds a measurement point at a first location in the displayed field of view that corresponds to a first location in the three-dimensional space. As the camera moves, the device displays the measurement point at a location in the displayed field of view that corresponds to the first location in the three-dimensional space. In response to a second touch input corresponding to a current location of the measurement point in the displayed field of view, the device enlarges display of the displayed field of view around the measurement point from the first magnification to a second, greater magnification.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Application Ser.No. 62/679,952, filed Jun. 3, 2018, and U.S. Provisional ApplicationSer. No. 62/668,249, filed May 7, 2018, which are incorporated byreference herein in their entireties.

TECHNICAL FIELD

This relates generally to electronic devices for virtual/augmentedreality, including but not limited to electronic devices for measuringphysical spaces and/or objects using virtual/augmented realityenvironments.

BACKGROUND

Augmented reality environments are useful for making measurements ofphysical spaces and objects therein, by providing a view of the physicalspace and enabling a user to superimpose measurements on the physicalspace and objects therein. But conventional methods of measuring usingaugmented reality are cumbersome, inefficient, and limited. In somecases, conventional methods of measuring using augmented reality arelimited in functionality. In some cases, conventional methods ofmeasuring using augmented reality require multiple separate inputs(e.g., a sequence of gestures and button presses, etc.) to achieve anintended outcome (e.g., through activation of numerous displayed userinterface elements to access different measurement functions). Inaddition, conventional methods take longer than necessary, therebywasting energy. This latter consideration is particularly important inbattery-operated devices.

SUMMARY

Accordingly, there is a need for computer systems with improved methodsand interfaces for measuring using virtual/augmented realityenvironments. Such methods and interfaces optionally complement orreplace conventional methods for measuring using virtual/augmentedreality environments. Such methods and interfaces reduce the number,extent, and/or nature of the inputs from a user and produce a moreefficient human-machine interface. For battery-operated devices, suchmethods and interfaces conserve power and increase the time betweenbattery charges.

The above deficiencies and other problems associated with userinterfaces for measuring using virtual/augmented reality are reduced oreliminated by the disclosed computer systems. In some embodiments, thecomputer system includes a desktop computer. In some embodiments, thecomputer system is portable (e.g., a notebook computer, tablet computer,or handheld device). In some embodiments, the computer system includes apersonal electronic device (e.g., a wearable electronic device, such asa watch). In some embodiments, the computer system has (and/or is incommunication with) a touchpad. In some embodiments, the computer systemhas (and/or is in communication with) a touch-sensitive display (alsoknown as a “touch screen” or “touch-screen display”). In someembodiments, the computer system has a graphical user interface (GUI),one or more processors, memory and one or more modules, programs or setsof instructions stored in the memory for performing multiple functions.In some embodiments, the user interacts with the GUI in part throughstylus and/or finger contacts and gestures on the touch-sensitivesurface. In some embodiments, in addition to an augmented reality-basedmeasurement function, the functions optionally include game playing,image editing, drawing, presenting, word processing, spreadsheet making,telephoning, video conferencing, e-mailing, instant messaging, workoutsupport, digital photographing, digital videoing, web browsing, digitalmusic playing, note taking, and/or digital video playing. Executableinstructions for performing these functions are, optionally, included ina non-transitory computer readable storage medium or other computerprogram product configured for execution by one or more processors.

In accordance with some embodiments, a method is performed at anelectronic device with a touch-sensitive display and one or morecameras. The method includes displaying, on the touch-sensitive display,a user interface of an application. The user interface includes arepresentation of a field of view of at least one of the one or morecameras. The representation of the field of view is displayed at a firstmagnification, and the representation of the field of view is updatedover time based on changes to current visual data detected by at leastone of the one or more cameras. The field of view includes at least aportion of a three-dimensional space. The method includes, whiledisplaying the representation of the field of view, detecting a firsttouch input on the touch-sensitive display, and, in response todetecting the first touch input, adding and displaying a measurementpoint at a first location in the representation of the field of viewthat corresponds to a first location in the three-dimensional space. Themethod also includes, after adding the measurement point and whilecontinuing to display the representation of the field of view: as atleast one of the one or more cameras moves, displaying the measurementpoint at a location in the representation of the field of view thatcorresponds to the first location in the three-dimensional space;detecting a second touch input at a location on the touch-sensitivedisplay that corresponds to a current location of the measurement pointin the representation of the field of view; and, in response todetecting the second touch input, enlarging display of at least aportion of the representation of the field of view from the firstmagnification to a second magnification, greater than the firstmagnification, wherein the enlarged display of the portion of therepresentation of the field of view includes the measurement point.

In accordance with some embodiments, a method is performed at anelectronic device with a touch-sensitive display, one or more sensors todetect intensities of contacts with the touch-sensitive display, and oneor more cameras. The method includes displaying, on the touch-sensitivedisplay, a user interface of an application. The user interface includesa representation of a field of view of at least one of the one or morecameras. The representation of the field of view is updated over timebased on changes to current visual data detected by at least one of theone or more cameras. The user interface also includes ameasurement-point-creation indicator that is displayed over therepresentation of the field of view. The field of view includes at leasta portion of a three-dimensional space. The method includes detecting acontact on the touch-sensitive display, and, while continuouslydetecting the contact on the touch-sensitive display, while themeasurement-point-creation indicator is displayed over a first locationin the representation of the field of view that corresponds to a firstlocation in the three-dimensional space, and in accordance with adetermination that first criteria are met, where the first criteriainclude a requirement that an intensity of the contact meet a respectiveintensity threshold in order for the first criteria to be met, addingand displaying a first measurement point in the representation of thefield of view that corresponds to the first location in thethree-dimensional space. The method also includes, after adding thefirst measurement point, updating the representation of the field ofview as the electronic device is moved. The method further includes,after the electronic device is moved, while themeasurement-point-creation indicator is displayed over a second locationin the representation of the field of view that corresponds to a secondlocation in the three-dimensional space, in accordance with adetermination that the first criteria are met while themeasurement-point-creation indicator is displayed over the secondlocation in the representation of the field of view that corresponds tothe second location in the three-dimensional space: adding anddisplaying a second measurement point in the representation of the fieldof view that corresponds to the second location in the three-dimensionalspace; and displaying a first measurement segment connecting the firstmeasurement point and the second measurement point.

In accordance with some embodiments, a method is performed at anelectronic device with a touch-sensitive display and one or morecameras. The method includes displaying, on the touch-sensitive display,a user interface of an application. The user interface includes arepresentation of a field of view of at least one of the one or morecameras. The representation of the field of view is updated over timebased on changes to current visual data detected by at least one of theone or more cameras. The user interface includes ameasurement-point-creation indicator that is displayed over therepresentation of the field of view. The field of view includes at leasta portion of a three-dimensional space. The method includes, whiledisplaying the representation of the field of view, determining ananchor point at a location in the representation of the field of viewthat corresponds to a first location in the three-dimensional space. Themethod also includes, as at least one of the one or more cameras move,while the measurement-point-creation indicator is over the anchor point,changing a visual appearance of the measurement-point-creation indicatorto indicate that a measurement point will be added at the anchor pointif a touch input meets first criteria. The method further includes,detecting a first touch input on the touch-sensitive display that meetsthe first criteria, and, in response to detecting the first touch inputthat meets the first criteria: in accordance with a determination thatthe measurement-point-creation indicator is over the anchor point whenthe first criteria are met, adding and displaying a first measurementpoint at the anchor point in the representation of the field of viewthat corresponds to the first location in the three-dimensional space;and in accordance with a determination that themeasurement-point-creation indicator is not over the anchor point whenthe first criteria are met, adding and displaying a first measurementpoint at a first location in the representation of the field of viewthat is away from the anchor point.

In accordance with some embodiments, a method is performed at anelectronic device with a display, an input device, and one or morecameras. The method includes displaying, on the display, a userinterface of an application. The user interface includes arepresentation of a field of view of at least one of the one or morecameras. The representation of the field of view is updated over timebased on changes to current visual data detected by at least one of theone or more cameras. The field of view includes a physical object in athree-dimensional space. The method includes, while displaying therepresentation of the field of view, detecting one or more user inputs,via the input device, that add, over the representation of the field ofview, a representation of a first measurement that corresponds to thephysical object. The method also includes concurrently displaying, overthe representation of the field of view, the representation of the firstmeasurement and a first label that describes the first measurement,where: in accordance with a determination that a first distance betweenthe electronic device and the physical object is less than a firstthreshold distance, the first label is displayed at a first thresholdsize; in accordance with a determination that the first distance betweenthe electronic device and the physical object is greater than a secondthreshold distance that is greater than the first threshold distance,the first label is displayed at a second threshold size that is smallerthan the first threshold size; and in accordance with a determinationthat the first distance between the electronic device and the physicalobject is between the first threshold distance and the second thresholddistance, the first label is displayed at a size, between the firstthreshold size and the second threshold size, that depends on the firstdistance between the electronic device and the physical object.

In accordance with some embodiments, a method is performed at anelectronic device with a display, an input device, and one or morecameras. The method includes displaying, on the display, a userinterface of an application. The user interface includes arepresentation of a field of view of at least one of the one or morecameras. The representation of the field of view is updated over timebased on changes to current visual data detected by at least one of theone or more cameras. The field of view includes a physical object in athree-dimensional space. The method includes, while displaying therepresentation of the field of view, detecting one or more user inputs,via the input device, that add, over the representation of the field ofview, a representation of a first measurement that corresponds to thephysical object, where the representation of the first measurementincludes a first endpoint that corresponds to a first location on thephysical object, the representation of the first measurement includes asecond endpoint that corresponds to a second location on the physicalobject; and the representation of the first measurement includes a firstline segment connecting the first endpoint and the second endpoint. Themethod also includes determining, based in part on the firstmeasurement, a first area in the representation of the field of viewthat adjoins the first line segment of the first measurement, where thefirst area corresponds to a physical rectangular area in thethree-dimensional space. The method further includes displaying anindication of the first area in the user interface, where the indicationis overlaid on the first area in the representation of the field ofview.

In accordance with some embodiments, a method is performed at anelectronic device with a touch-sensitive display and one or morecameras. The method includes displaying, on the touch-sensitive display,a first user interface of an application. The first user interfaceincludes a representation of a field of view of at least one of the oneor more cameras. The representation of the field of view is updated overtime based on changes to current visual data detected by at least one ofthe one or more cameras. The field of view includes a physical object ina three-dimensional space. A representation of a measurement of thephysical object is superimposed on an image of the physical object inthe representation of the field of view. The method includes, whiledisplaying the first user interface, detecting a first touch input onthe touch-sensitive display on the representation of the measurement.The method further includes, in response to detecting the first touchinput on the touch-sensitive display on the representation of themeasurement, initiating a process for sharing information about themeasurement.

In accordance with some embodiments, a method is performed at anelectronic device with a display, an input device, and one or morecameras. The method includes displaying, on the display, a userinterface of an application. The user interface includes arepresentation of a field of view of at least one of the one or morecameras. The representation of the field of view is updated over timebased on changes to current visual data detected by at least one of theone or more cameras. The field of view includes at least a portion of athree-dimensional space. The method includes detecting movement of theelectronic device that moves the field of view of at least one of theone or more cameras in a first direction. The method also includes,while detecting the movement of the electronic device that moves thefield of view in the first direction: updating the representation of thefield of view in accordance with the movement of the electronic device;identifying one or more first elements in the representation of thefield of view that extend along the first direction; and, based at leastin part on the determination of the one or more first elements,displaying, in the representation of the field of view, a first guidethat extends in the first direction and that corresponds to one of theone or more first identified elements.

In accordance with some embodiments, a method is performed at anelectronic device with one or more input devices, one or more displaydevices, and one or more cameras: The method includes displaying, viathe one or more display devices, a user interface that includes arepresentation of a physical space. The method includes, whiledisplaying the representation of the physical space, receiving a firstset of one or more inputs to create a virtual annotation in therepresentation of the physical space. The method also includes, inresponse to receiving the first set of one or more inputs, adding afirst virtual annotation to the representation of the physical space.The first virtual annotation is linked to a portion of therepresentation of the physical space. The method also includes, afteradding the first virtual annotation to the representation of thephysical space, receiving a second set of one or more inputs associatedwith the representation of the physical space. The method furtherincludes, in response to receiving the second set of one or more inputsassociated with the representation of the physical space: in accordancewith a determination that the second set of one or more inputscorresponds to a request to create a virtual annotation in therepresentation of the physical space that is within a threshold distancefrom the first virtual annotation, creating a second virtual annotationin the representation of the physical space while maintaining the firstvirtual annotation in the representation of the physical space; and, inaccordance with a determination that the second set of one or moreinputs corresponds to a request to create a virtual annotation in therepresentation of the physical space that is outside of the thresholddistance from the first virtual annotation, creating a second virtualannotation in the representation of the physical space and removing thefirst virtual annotation from the representation of the physical space.

In accordance with some embodiments, a method is performed at anelectronic device with one or more input devices, one or more displaydevices, and one or more cameras. The method includes displaying, viathe one or more display devices, an annotation placement user interface.The annotation placement user interface includes: a representation of aphysical space; and a placement user interface element that indicates alocation at which a virtual annotation will be placed in therepresentation of the physical space in response to detecting anannotation placement input. The method includes, while displaying theannotation placement user interface, detecting movement of at least oneof the one or more cameras relative to the physical space. The movementof at least one of the one or more cameras starts while the placementuser interface element is displayed at a location in the representationof the physical space that corresponds to a first portion of thephysical space. The method includes, in response to detecting themovement of at least one of the one or more cameras relative to thephysical space, moving the placement user interface element to alocation in the representation of the physical space that corresponds toa second portion of the physical space that is different from the firstportion of the physical space, and updating an appearance of theannotation placement user interface in accordance with the movement ofat least one of the one or more cameras relative to the physical space,including: in accordance with a determination that the electronic deviceis unable to identify an object in the second portion of the physicalspace whose corresponding object in the representation of the physicalspace can be linked to a virtual annotation, ceasing to display at leasta portion of the placement user interface element; and in accordancewith a determination that the device has identified an object in thesecond portion of the physical space whose corresponding object in therepresentation of the physical space can be linked to a virtualannotation, maintaining display of the placement user interface element.

In accordance with some embodiments, a computer system (e.g., anelectronic device) includes (and/or is in communication with) a displaygeneration component (e.g., a display, a projector, a heads-up display,or the like), one or more cameras (e.g., video cameras that continuouslyprovide a live preview of at least a portion of the contents that arewithin the field of view of the cameras and optionally generate videooutputs including one or more streams of image frames capturing thecontents within the field of view of the cameras), and one or more inputdevices (e.g., a touch-sensitive surface, such as a touch-sensitiveremote control, or a touch-screen display that also serves as thedisplay generation component, a mouse, a joystick, a wand controller,and/or cameras tracking the position of one or more features of the usersuch as the user's hands), optionally one or more attitude sensors,optionally one or more sensors to detect intensities of contacts withthe touch-sensitive surface, optionally one or more tactile outputgenerators, one or more processors, and memory storing one or moreprograms; the one or more programs are configured to be executed by theone or more processors and the one or more programs include instructionsfor performing or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, acomputer readable storage medium has stored therein instructions, which,when executed by a computer system that includes (and/or is incommunication with) a display generation component, one or more cameras,one or more input devices, optionally one or more attitude sensors,optionally one or more sensors to detect intensities of contacts withthe touch-sensitive surface, and optionally one or more tactile outputgenerators, cause the computer system to perform or cause performance ofthe operations of any of the methods described herein. In accordancewith some embodiments, a graphical user interface on a computer systemthat includes (and/or is in communication with) a display generationcomponent, one or more cameras, one or more input devices, optionallyone or more attitude sensors, optionally one or more sensors to detectintensities of contacts with the touch-sensitive surface, optionally oneor more tactile output generators, a memory, and one or more processorsto execute one or more programs stored in the memory includes one ormore of the elements displayed in any of the methods described herein,which are updated in response to inputs, as described in any of themethods described herein. In accordance with some embodiments, acomputer system includes (and/or is in communication with) a displaygeneration component, one or more cameras, one or more input devices,optionally one or more attitude sensors, optionally one or more sensorsto detect intensities of contacts with the touch-sensitive surface,optionally one or more tactile output generators, and means forperforming or causing performance of the operations of any of themethods described herein. In accordance with some embodiments, aninformation processing apparatus, for use in a computer system thatincludes (and/or is in communication with) a display generationcomponent, one or more cameras, one or more input devices, optionallyone or more attitude sensors, optionally one or more sensors to detectintensities of contacts with the touch-sensitive surface, and optionallyone or more tactile output generators, includes means for performing orcausing performance of the operations of any of the methods describedherein.

Thus, computer systems that have (and/or are in communication with) adisplay generation component, one or more cameras, one or more inputdevices, optionally one or more attitude sensors, optionally one or moresensors to detect intensities of contacts with the touch-sensitivesurface, and optionally one or more tactile output generators, areprovided with improved methods and interfaces for measuring physicalobjects using virtual/augmented reality environments, thereby increasingthe effectiveness, efficiency, and user satisfaction with such computersystems. Such methods and interfaces may complement or replaceconventional methods for measuring physical objects usingvirtual/augmented reality environments.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the various described embodiments,reference should be made to the Description of Embodiments below, inconjunction with the following drawings in which like reference numeralsrefer to corresponding parts throughout the figures.

FIG. 1A is a block diagram illustrating a portable multifunction devicewith a touch-sensitive display in accordance with some embodiments.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments.

FIG. 2 illustrates a portable multifunction device having a touch screenin accordance with some embodiments.

FIG. 3A is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments.

FIGS. 3B-3C are block diagrams of example computer systems in accordancewith some embodiments.

FIG. 4A illustrates an example user interface for a menu of applicationson a portable multifunction device in accordance with some embodiments.

FIG. 4B illustrates an example user interface for a multifunction devicewith a touch-sensitive surface that is separate from the display inaccordance with some embodiments.

FIGS. 4C-4E illustrate examples of dynamic intensity thresholds inaccordance with some embodiments.

FIGS. 4F-4K illustrate a set of sample tactile output patterns inaccordance with some embodiments.

FIGS. 5A-5CO illustrate example user interfaces for making measurementsof a physical space using an augmented reality environment in accordancewith some embodiments.

FIGS. 6A-6C are flow diagrams of a process for interacting with anapplication for making measurements of a physical space using anaugmented reality environment in accordance with some embodiments.

FIGS. 7A-7E are flow diagrams of a process for adding measurements to adisplayed representation of a physical space in an augmented realityenvironment in accordance with some embodiments.

FIGS. 8A-8C are flow diagrams of a process for adding virtualmeasurement points at automatically determined anchor points in anaugmented reality environment in accordance with some embodiments.

FIGS. 9A-9B are flow diagrams of a process for displaying labels formeasurements of a physical space in an augmented reality environment inaccordance with some embodiments.

FIGS. 10A-10B are flow diagrams of a process for measuring andinteracting with rectangular areas in a physical space in an augmentedreality environment in accordance with some embodiments.

FIGS. 11A-11B are flow diagrams of a process for interacting with andmanaging measurement information in an augmented reality environment inaccordance with some embodiments.

FIGS. 12A-12C are flow diagrams of a process for providing automaticallydetermined alignment guides in an augmented reality environment inaccordance with some embodiments.

FIGS. 13A-13C are flow diagrams of a process for automatically removingpreviously-added virtual annotations in an augmented reality environmentin accordance with some embodiments.

FIGS. 14A-14D are flow diagrams of a process for indicating whetherobjects in a physical space have been identified as objects whosecorresponding representations in an augmented reality environment can betracked in accordance with some embodiments.

DESCRIPTION OF EMBODIMENTS

As noted above, augmented reality environments are useful for makingmeasurements of physical spaces and objects therein, by providing a viewof the physical space and enabling a user to superimpose measurements onthe physical space and physical objects therein. Conventional methods ofmeasuring with augmented reality environments are often limited infunctionality. In some cases, conventional methods require multipleseparate inputs (e.g., a sequence of gestures and button presses, etc.)to achieve an intended outcome (e.g., through activation of numerousdisplayed user interface elements to access different measurementfunctions). The embodiments disclosed herein provide an intuitive wayfor a user to make measurements with an augmented reality environment(e.g., by enabling the user to perform different operations in theaugmented reality environment with fewer inputs, and/or by simplifyingthe user interface). Additionally, the embodiments herein provideimproved visual and tactile feedback that provide additional informationto the user about the physical objects being measured and about theoperations being performed in the augmented reality environment.

The systems, methods, and GUIs described herein improve user interfaceinteractions with virtual/augmented reality environments in multipleways. For example, they make it easier to measure features in a physicalspace using an augmented reality environment, by providing automaticdetection of features in the physical space, improved labeling, andalignment guides (e.g., for improved measurement point placement andarea recognition), and by enabling the user to interact with and managemeasurement information.

Below, FIGS. 1A-1B, 2, and 3A-3C provide a description of exampledevices. FIGS. 4A-4B and 5A-5CO illustrate example contexts and exampleuser interfaces for making measurements of a physical space using anaugmented reality environment. FIGS. 6A-6C illustrate a flow diagram ofa method of interacting with an application for making measurements of aphysical space using an augmented reality environment. FIGS. 7A-7Eillustrate a flow diagram of a method of adding measurements to adisplayed representation of a physical space in an augmented realityenvironment. FIGS. 8A-8C illustrate a flow diagram of a method of addingvirtual measurement points at automatically determined anchor points inan augmented reality environment. FIGS. 9A-9B illustrate a flow diagramof a method of displaying labels for measurements of a physical space inan augmented reality environment. FIGS. 10A-10B illustrate a flowdiagram of a method of measuring and interacting with rectangular areasin a physical space in an augmented reality environment. FIGS. 11A-11Billustrate a flow diagram of a method of interacting with and managingmeasurement information in an augmented reality environment. FIGS.12A-12C illustrate a flow diagram of a method of providing automaticallydetermined alignment guides in an augmented reality environment. FIGS.13A-13C are flow diagrams of a process for automatically removingpreviously-added virtual annotations in an augmented realityenvironment. FIGS. 14A-14D are flow diagrams of a process for indicatingwhether objects in a physical space have been identified as objectswhose corresponding representations in an augmented reality environmentcan be tracked. The user interfaces in FIGS. 5A-5CO are used toillustrate the processes in FIGS. 6A-6C, 7A-7E, 8A-8C, 9A-9B, 10A-10B,11A-11B, 12A-12C, 13A-13C, and 14A-14D.

Example Devices

Reference will now be made in detail to embodiments, examples of whichare illustrated in the accompanying drawings. In the following detaileddescription, numerous specific details are set forth in order to providea thorough understanding of the various described embodiments. However,it will be apparent to one of ordinary skill in the art that the variousdescribed embodiments may be practiced without these specific details.In other instances, well-known methods, procedures, components,circuits, and networks have not been described in detail so as not tounnecessarily obscure aspects of the embodiments.

It will also be understood that, although the terms first, second, etc.are, in some instances, used herein to describe various elements, theseelements should not be limited by these terms. These terms are only usedto distinguish one element from another. For example, a first contactcould be termed a second contact, and, similarly, a second contact couldbe termed a first contact, without departing from the scope of thevarious described embodiments. The first contact and the second contactare both contacts, but they are not the same contact, unless the contextclearly indicates otherwise.

The terminology used in the description of the various describedembodiments herein is for the purpose of describing particularembodiments only and is not intended to be limiting. As used in thedescription of the various described embodiments and the appendedclaims, the singular forms “a,” “an,” and “the” are intended to includethe plural forms as well, unless the context clearly indicatesotherwise. It will also be understood that the term “and/or” as usedherein refers to and encompasses any and all possible combinations ofone or more of the associated listed items. It will be furtherunderstood that the terms “includes,” “including,” “comprises,” and/or“comprising,” when used in this specification, specify the presence ofstated features, integers, steps, operations, elements, and/orcomponents, but do not preclude the presence or addition of one or moreother features, integers, steps, operations, elements, components,and/or groups thereof.

As used herein, the term “if” is, optionally, construed to mean “when”or “upon” or “in response to determining” or “in response to detecting,”depending on the context. Similarly, the phrase “if it is determined” or“if [a stated condition or event] is detected” is, optionally, construedto mean “upon determining” or “in response to determining” or “upondetecting [the stated condition or event]” or “in response to detecting[the stated condition or event],” depending on the context.

Computer systems for virtual/augmented reality include electronicdevices that produce virtual/augmented reality environments. Embodimentsof electronic devices, user interfaces for such devices, and associatedprocesses for using such devices are described. In some embodiments, thedevice is a portable communications device, such as a mobile telephone,that also contains other functions, such as PDA and/or music playerfunctions. Example embodiments of portable multifunction devicesinclude, without limitation, the iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. Other portable electronic devices,such as laptops or tablet computers with touch-sensitive surfaces (e.g.,touch-screen displays and/or touchpads), are, optionally, used. Itshould also be understood that, in some embodiments, the device is not aportable communications device, but is a desktop computer with atouch-sensitive surface (e.g., a touch-screen display and/or a touchpad)that also includes, or is in communication with, one or more cameras.

In the discussion that follows, a computer system that includes anelectronic device that has (and/or is in communication with) a displayand a touch-sensitive surface is described. It should be understood,however, that the computer system optionally includes one or more otherphysical user-interface devices, such as a physical keyboard, a mouse, ajoystick, a wand controller, and/or cameras tracking the position of oneor more features of the user such as the user's hands.

The device typically supports a variety of applications, such as one ormore of the following: a gaming application, a note taking application,a drawing application, a presentation application, a word processingapplication, a spreadsheet application, a telephone application, a videoconferencing application, an e-mail application, an instant messagingapplication, a workout support application, a photo managementapplication, a digital camera application, a digital video cameraapplication, a web browsing application, a digital music playerapplication, and/or a digital video player application.

The various applications that are executed on the device optionally useat least one common physical user-interface device, such as thetouch-sensitive surface. One or more functions of the touch-sensitivesurface as well as corresponding information displayed by the deviceare, optionally, adjusted and/or varied from one application to the nextand/or within a respective application. In this way, a common physicalarchitecture (such as the touch-sensitive surface) of the deviceoptionally supports the variety of applications with user interfacesthat are intuitive and transparent to the user.

Attention is now directed toward embodiments of portable devices withtouch-sensitive displays. FIG. 1A is a block diagram illustratingportable multifunction device 100 with touch-sensitive display system112 in accordance with some embodiments. Touch-sensitive display system112 is sometimes called a “touch screen” for convenience, and issometimes simply called a touch-sensitive display. Device 100 includesmemory 102 (which optionally includes one or more computer readablestorage mediums), memory controller 122, one or more processing units(CPUs) 120, peripherals interface 118, RF circuitry 108, audio circuitry110, speaker 111, microphone 113, input/output (I/O) subsystem 106,other input or control devices 116, and external port 124. Device 100optionally includes one or more optical sensors 164 (e.g., as part ofone or more cameras). Device 100 optionally includes one or moreintensity sensors 165 for detecting intensities of contacts on device100 (e.g., a touch-sensitive surface such as touch-sensitive displaysystem 112 of device 100). Device 100 optionally includes one or moretactile output generators 163 for generating tactile outputs on device100 (e.g., generating tactile outputs on a touch-sensitive surface suchas touch-sensitive display system 112 of device 100 or touchpad 355 ofdevice 300). These components optionally communicate over one or morecommunication buses or signal lines 103.

As used in the specification and claims, the term “tactile output”refers to physical displacement of a device relative to a previousposition of the device, physical displacement of a component (e.g., atouch-sensitive surface) of a device relative to another component(e.g., housing) of the device, or displacement of the component relativeto a center of mass of the device that will be detected by a user withthe user's sense of touch. For example, in situations where the deviceor the component of the device is in contact with a surface of a userthat is sensitive to touch (e.g., a finger, palm, or other part of auser's hand), the tactile output generated by the physical displacementwill be interpreted by the user as a tactile sensation corresponding toa perceived change in physical characteristics of the device or thecomponent of the device. For example, movement of a touch-sensitivesurface (e.g., a touch-sensitive display or trackpad) is, optionally,interpreted by the user as a “down click” or “up click” of a physicalactuator button. In some cases, a user will feel a tactile sensationsuch as an “down click” or “up click” even when there is no movement ofa physical actuator button associated with the touch-sensitive surfacethat is physically pressed (e.g., displaced) by the user's movements. Asanother example, movement of the touch-sensitive surface is, optionally,interpreted or sensed by the user as “roughness” of the touch-sensitivesurface, even when there is no change in smoothness of thetouch-sensitive surface. While such interpretations of touch by a userwill be subject to the individualized sensory perceptions of the user,there are many sensory perceptions of touch that are common to a largemajority of users. Thus, when a tactile output is described ascorresponding to a particular sensory perception of a user (e.g., an “upclick,” a “down click,” “roughness”), unless otherwise stated, thegenerated tactile output corresponds to physical displacement of thedevice or a component thereof that will generate the described sensoryperception for a typical (or average) user. Using tactile outputs toprovide haptic feedback to a user enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device) which, additionally, reducespower usage and improves battery life of the device by enabling the userto use the device more quickly and efficiently.

In some embodiments, a tactile output pattern specifies characteristicsof a tactile output, such as the amplitude of the tactile output, theshape of a movement waveform of the tactile output, the frequency of thetactile output, and/or the duration of the tactile output.

When tactile outputs with different tactile output patterns aregenerated by a device (e.g., via one or more tactile output generatorsthat move a moveable mass to generate tactile outputs), the tactileoutputs may invoke different haptic sensations in a user holding ortouching the device. While the sensation of the user is based on theuser's perception of the tactile output, most users will be able toidentify changes in waveform, frequency, and amplitude of tactileoutputs generated by the device. Thus, the waveform, frequency andamplitude can be adjusted to indicate to the user that differentoperations have been performed. As such, tactile outputs with tactileoutput patterns that are designed, selected, and/or engineered tosimulate characteristics (e.g., size, material, weight, stiffness,smoothness, etc.); behaviors (e.g., oscillation, displacement,acceleration, rotation, expansion, etc.); and/or interactions (e.g.,collision, adhesion, repulsion, attraction, friction, etc.) of objectsin a given environment (e.g., a user interface that includes graphicalfeatures and objects, a simulated physical environment with virtualboundaries and virtual objects, a real physical environment withphysical boundaries and physical objects, and/or a combination of any ofthe above) will, in some circumstances, provide helpful feedback tousers that reduces input errors and increases the efficiency of theuser's operation of the device. Additionally, tactile outputs are,optionally, generated to correspond to feedback that is unrelated to asimulated physical characteristic, such as an input threshold or aselection of an object. Such tactile outputs will, in somecircumstances, provide helpful feedback to users that reduces inputerrors and increases the efficiency of the user's operation of thedevice.

In some embodiments, a tactile output with a suitable tactile outputpattern serves as a cue for the occurrence of an event of interest in auser interface or behind the scenes in a device. Examples of the eventsof interest include activation of an affordance (e.g., a real or virtualbutton, or toggle switch) provided on the device or in a user interface,success or failure of a requested operation, reaching or crossing aboundary in a user interface, entry into a new state, switching of inputfocus between objects, activation of a new mode, reaching or crossing aninput threshold, detection or recognition of a type of input or gesture,etc. In some embodiments, tactile outputs are provided to serve as awarning or an alert for an impending event or outcome that would occurunless a redirection or interruption input is timely detected. Tactileoutputs are also used in other contexts to enrich the user experience,improve the accessibility of the device to users with visual or motordifficulties or other accessibility needs, and/or improve efficiency andfunctionality of the user interface and/or the device. Tactile outputsare optionally accompanied with audio outputs and/or visible userinterface changes, which further enhance a user's experience when theuser interacts with a user interface and/or the device, and facilitatebetter conveyance of information regarding the state of the userinterface and/or the device, and which reduce input errors and increasethe efficiency of the user's operation of the device.

FIGS. 4F-4H provide a set of sample tactile output patterns that may beused, either individually or in combination, either as is or through oneor more transformations (e.g., modulation, amplification, truncation,etc.), to create suitable haptic feedback in various scenarios and forvarious purposes, such as those mentioned above and those described withrespect to the user interfaces and methods discussed herein. Thisexample of a palette of tactile outputs shows how a set of threewaveforms and eight frequencies can be used to produce an array oftactile output patterns. In addition to the tactile output patternsshown in this figure, each of these tactile output patterns isoptionally adjusted in amplitude by changing a gain value for thetactile output pattern, as shown, for example for FullTap 80 Hz, FullTap200 Hz, MiniTap 80 Hz, MiniTap 200 Hz, MicroTap 80 Hz, and MicroTap 200Hz in FIGS. 41-4K, which are each shown with variants having a gain of1.0, 0.75, 0.5, and 0.25. As shown in FIGS. 41-4K, changing the gain ofa tactile output pattern changes the amplitude of the pattern withoutchanging the frequency of the pattern or changing the shape of thewaveform. In some embodiments, changing the frequency of a tactileoutput pattern also results in a lower amplitude as some tactile outputgenerators are limited by how much force can be applied to the moveablemass and thus higher frequency movements of the mass are constrained tolower amplitudes to ensure that the acceleration needed to create thewaveform does not require force outside of an operational force range ofthe tactile output generator (e.g., the peak amplitudes of the FullTapat 230 Hz, 270 Hz, and 300 Hz are lower than the amplitudes of theFullTap at 80 Hz, 100 Hz, 125 Hz, and 200 Hz).

FIGS. 4F-4K show tactile output patterns that have a particularwaveform. The waveform of a tactile output pattern represents thepattern of physical displacements relative to a neutral position (e.g.,xzero) versus time that a moveable mass goes through to generate atactile output with that tactile output pattern. For example, a firstset of tactile output patterns shown in FIG. 4F (e.g., tactile outputpatterns of a “FullTap”) each have a waveform that includes anoscillation with two complete cycles (e.g., an oscillation that startsand ends in a neutral position and crosses the neutral position threetimes). A second set of tactile output patterns shown in FIG. 4G (e.g.,tactile output patterns of a “MiniTap”) each have a waveform thatincludes an oscillation that includes one complete cycle (e.g., anoscillation that starts and ends in a neutral position and crosses theneutral position one time). A third set of tactile output patterns shownin FIG. 4H (e.g., tactile output patterns of a “MicroTap”) each have awaveform that includes an oscillation that include one half of acomplete cycle (e.g., an oscillation that starts and ends in a neutralposition and does not cross the neutral position). The waveform of atactile output pattern also includes a start buffer and an end bufferthat represent the gradual speeding up and slowing down of the moveablemass at the start and at the end of the tactile output. The examplewaveforms shown in FIGS. 4F-4K include xmin and xmax values whichrepresent the maximum and minimum extent of movement of the moveablemass. For larger electronic devices with larger moveable masses, theremay be larger or smaller minimum and maximum extents of movement of themass. The examples shown in FIGS. 4F-4K describe movement of a mass in 1dimension, however similar principles would also apply to movement of amoveable mass in two or three dimensions.

As shown in FIGS. 4F-4H, each tactile output pattern also has acorresponding characteristic frequency that affects the “pitch” of ahaptic sensation that is felt by a user from a tactile output with thatcharacteristic frequency. For a continuous tactile output, thecharacteristic frequency represents the number of cycles that arecompleted within a given period of time (e.g., cycles per second) by themoveable mass of the tactile output generator. For a discrete tactileoutput, a discrete output signal (e.g., with 0.5, 1, or 2 cycles) isgenerated, and the characteristic frequency value specifies how fast themoveable mass needs to move to generate a tactile output with thatcharacteristic frequency. As shown in FIGS. 4F-4H, for each type oftactile output (e.g., as defined by a respective waveform, such asFullTap, MiniTap, or MicroTap), a higher frequency value corresponds tofaster movement(s) by the moveable mass, and hence, in general, ashorter time to complete the tactile output (e.g., including the time tocomplete the required number of cycle(s) for the discrete tactileoutput, plus a start and an end buffer time). For example, a FullTapwith a characteristic frequency of 80 Hz takes longer to complete thanFullTap with a characteristic frequency of 100 Hz (e.g., 35.4 ms vs.28.3 ms in FIG. 4F). In addition, for a given frequency, a tactileoutput with more cycles in its waveform at a respective frequency takeslonger to complete than a tactile output with fewer cycles its waveformat the same respective frequency. For example, a FullTap at 150 Hz takeslonger to complete than a MiniTap at 150 Hz (e.g., 19.4 ms vs. 12.8 ms),and a MiniTap at 150 Hz takes longer to complete than a MicroTap at 150Hz (e.g., 12.8 ms vs. 9.4 ms). However, for tactile output patterns withdifferent frequencies this rule may not apply (e.g., tactile outputswith more cycles but a higher frequency may take a shorter amount oftime to complete than tactile outputs with fewer cycles but a lowerfrequency, and vice versa). For example, at 300 Hz, a FullTap takes aslong as a MiniTap (e.g., 9.9 ms).

As shown in FIGS. 4F-4H, a tactile output pattern also has acharacteristic amplitude that affects the amount of energy that iscontained in a tactile signal, or a “strength” of a haptic sensationthat may be felt by a user through a tactile output with thatcharacteristic amplitude. In some embodiments, the characteristicamplitude of a tactile output pattern refers to an absolute ornormalized value that represents the maximum displacement of themoveable mass from a neutral position when generating the tactileoutput. In some embodiments, the characteristic amplitude of a tactileoutput pattern is adjustable, e.g., by a fixed or dynamically determinedgain factor (e.g., a value between 0 and 1), in accordance with variousconditions (e.g., customized based on user interface contexts andbehaviors) and/or preconfigured metrics (e.g., input-based metrics,and/or user-interface-based metrics). In some embodiments, aninput-based metric (e.g., an intensity-change metric or an input-speedmetric) measures a characteristic of an input (e.g., a rate of change ofa characteristic intensity of a contact in a press input or a rate ofmovement of the contact across a touch-sensitive surface) during theinput that triggers generation of a tactile output. In some embodiments,a user-interface-based metric (e.g., a speed-across-boundary metric)measures a characteristic of a user interface element (e.g., a speed ofmovement of the element across a hidden or visible boundary in a userinterface) during the user interface change that triggers generation ofthe tactile output. In some embodiments, the characteristic amplitude ofa tactile output pattern may be modulated by an “envelope” and the peaksof adjacent cycles may have different amplitudes, where one of thewaveforms shown above is further modified by multiplication by anenvelope parameter that changes over time (e.g., from 0 to 1) togradually adjust amplitude of portions of the tactile output over timeas the tactile output is being generated.

Although specific frequencies, amplitudes, and waveforms are representedin the sample tactile output patterns in FIGS. 4F-4H for illustrativepurposes, tactile output patterns with other frequencies, amplitudes,and waveforms may be used for similar purposes. For example, waveformsthat have between 0.5 to 4 cycles can be used. Other frequencies in therange of 60 Hz-400 Hz may be used as well.

It should be appreciated that device 100 is only one example of aportable multifunction device, and that device 100 optionally has moreor fewer components than shown, optionally combines two or morecomponents, or optionally has a different configuration or arrangementof the components. The various components shown in FIG. 1A areimplemented in hardware, software, firmware, or a combination thereof,including one or more signal processing and/or application specificintegrated circuits.

Memory 102 optionally includes high-speed random access memory andoptionally also includes non-volatile memory, such as one or moremagnetic disk storage devices, flash memory devices, or othernon-volatile solid-state memory devices. Access to memory 102 by othercomponents of device 100, such as CPU(s) 120 and the peripheralsinterface 118, is, optionally, controlled by memory controller 122.

Peripherals interface 118 can be used to couple input and outputperipherals of the device to CPU(s) 120 and memory 102. The one or moreprocessors 120 run or execute various software programs and/or sets ofinstructions stored in memory 102 to perform various functions fordevice 100 and to process data.

In some embodiments, peripherals interface 118, CPU(s) 120, and memorycontroller 122 are, optionally, implemented on a single chip, such aschip 104. In some other embodiments, they are, optionally, implementedon separate chips.

RF (radio frequency) circuitry 108 receives and sends RF signals, alsocalled electromagnetic signals. RF circuitry 108 converts electricalsignals to/from electromagnetic signals and communicates withcommunications networks and other communications devices via theelectromagnetic signals. RF circuitry 108 optionally includes well-knowncircuitry for performing these functions, including but not limited toan antenna system, an RF transceiver, one or more amplifiers, a tuner,one or more oscillators, a digital signal processor, a CODEC chipset, asubscriber identity module (SIM) card, memory, and so forth. RFcircuitry 108 optionally communicates with networks, such as theInternet, also referred to as the World Wide Web (WWW), an intranetand/or a wireless network, such as a cellular telephone network, awireless local area network (LAN) and/or a metropolitan area network(MAN), and other devices by wireless communication. The wirelesscommunication optionally uses any of a plurality of communicationsstandards, protocols and technologies, including but not limited toGlobal System for Mobile Communications (GSM), Enhanced Data GSMEnvironment (EDGE), high-speed downlink packet access (HSDPA),high-speed uplink packet access (HSUPA), Evolution, Data-Only (EV-DO),HSPA, HSPA+, Dual-Cell HSPA (DC-HSPA), long term evolution (LTE), nearfield communication (NFC), wideband code division multiple access(W-CDMA), code division multiple access (CDMA), time division multipleaccess (TDMA), Bluetooth, Wireless Fidelity (Wi-Fi) (e.g., IEEE 802.11a,IEEE 802.11ac, IEEE 802.11ax, IEEE 802.11b, IEEE 802.11g and/or IEEE802.11n), voice over Internet Protocol (VoIP), Wi-MAX, a protocol fore-mail (e.g., Internet message access protocol (IMAP) and/or post officeprotocol (POP)), instant messaging (e.g., extensible messaging andpresence protocol (XMPP), Session Initiation Protocol for InstantMessaging and Presence Leveraging Extensions (SIMPLE), Instant Messagingand Presence Service (IMPS)), and/or Short Message Service (SMS), or anyother suitable communication protocol, including communication protocolsnot yet developed as of the filing date of this document.

Audio circuitry 110, speaker 111, and microphone 113 provide an audiointerface between a user and device 100. Audio circuitry 110 receivesaudio data from peripherals interface 118, converts the audio data to anelectrical signal, and transmits the electrical signal to speaker 111.Speaker 111 converts the electrical signal to human-audible sound waves.Audio circuitry 110 also receives electrical signals converted bymicrophone 113 from sound waves. Audio circuitry 110 converts theelectrical signal to audio data and transmits the audio data toperipherals interface 118 for processing. Audio data is, optionally,retrieved from and/or transmitted to memory 102 and/or RF circuitry 108by peripherals interface 118. In some embodiments, audio circuitry 110also includes a headset jack (e.g., 212, FIG. 2). The headset jackprovides an interface between audio circuitry 110 and removable audioinput/output peripherals, such as output-only headphones or a headsetwith both output (e.g., a headphone for one or both ears) and input(e.g., a microphone).

I/O subsystem 106 couples input/output peripherals on device 100, suchas touch-sensitive display system 112 and other input or control devices116, with peripherals interface 118. I/O subsystem 106 optionallyincludes display controller 156, optical sensor controller 158,intensity sensor controller 159, haptic feedback controller 161, and oneor more input controllers 160 for other input or control devices. Theone or more input controllers 160 receive/send electrical signalsfrom/to other input or control devices 116. The other input or controldevices 116 optionally include physical buttons (e.g., push buttons,rocker buttons, etc.), dials, slider switches, joysticks, click wheels,and so forth. In some alternate embodiments, input controller(s) 160are, optionally, coupled with any (or none) of the following: akeyboard, infrared port, USB port, stylus, and/or a pointer device suchas a mouse. The one or more buttons (e.g., 208, FIG. 2) optionallyinclude an up/down button for volume control of speaker 111 and/ormicrophone 113. The one or more buttons optionally include a push button(e.g., 206, FIG. 2).

Touch-sensitive display system 112 provides an input interface and anoutput interface between the device and a user. Display controller 156receives and/or sends electrical signals from/to touch-sensitive displaysystem 112. Touch-sensitive display system 112 displays visual output tothe user. The visual output optionally includes graphics, text, icons,video, and any combination thereof (collectively termed “graphics”). Insome embodiments, some or all of the visual output corresponds to userinterface objects. As used herein, the term “affordance” refers to auser-interactive graphical user interface object (e.g., a graphical userinterface object that is configured to respond to inputs directed towardthe graphical user interface object). Examples of user-interactivegraphical user interface objects include, without limitation, a button,slider, icon, selectable menu item, switch, hyperlink, or other userinterface control.

Touch-sensitive display system 112 has a touch-sensitive surface, sensoror set of sensors that accepts input from the user based on hapticand/or tactile contact. Touch-sensitive display system 112 and displaycontroller 156 (along with any associated modules and/or sets ofinstructions in memory 102) detect contact (and any movement or breakingof the contact) on touch-sensitive display system 112 and converts thedetected contact into interaction with user-interface objects (e.g., oneor more soft keys, icons, web pages or images) that are displayed ontouch-sensitive display system 112. In some embodiments, a point ofcontact between touch-sensitive display system 112 and the usercorresponds to a finger of the user or a stylus.

Touch-sensitive display system 112 optionally uses LCD (liquid crystaldisplay) technology, LPD (light emitting polymer display) technology, orLED (light emitting diode) technology, although other displaytechnologies are used in other embodiments. Touch-sensitive displaysystem 112 and display controller 156 optionally detect contact and anymovement or breaking thereof using any of a plurality of touch sensingtechnologies now known or later developed, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch-sensitive displaysystem 112. In some embodiments, projected mutual capacitance sensingtechnology is used, such as that found in the iPhone®, iPod Touch®, andiPad® from Apple Inc. of Cupertino, Calif.

Touch-sensitive display system 112 optionally has a video resolution inexcess of 100 dpi. In some embodiments, the touch screen videoresolution is in excess of 400 dpi (e.g., 500 dpi, 800 dpi, or greater).The user optionally makes contact with touch-sensitive display system112 using any suitable object or appendage, such as a stylus, a finger,and so forth. In some embodiments, the user interface is designed towork with finger-based contacts and gestures, which can be less precisethan stylus-based input due to the larger area of contact of a finger onthe touch screen. In some embodiments, the device translates the roughfinger-based input into a precise pointer/cursor position or command forperforming the actions desired by the user.

In some embodiments, in addition to the touch screen, device 100optionally includes a touchpad (not shown) for activating ordeactivating particular functions. In some embodiments, the touchpad isa touch-sensitive area of the device that, unlike the touch screen, doesnot display visual output. The touchpad is, optionally, atouch-sensitive surface that is separate from touch-sensitive displaysystem 112 or an extension of the touch-sensitive surface formed by thetouch screen.

Device 100 also includes power system 162 for powering the variouscomponents. Power system 162 optionally includes a power managementsystem, one or more power sources (e.g., battery, alternating current(AC)), a recharging system, a power failure detection circuit, a powerconverter or inverter, a power status indicator (e.g., a light-emittingdiode (LED)) and any other components associated with the generation,management and distribution of power in portable devices.

Device 100 optionally also includes one or more optical sensors 164(e.g., as part of one or more cameras). FIG. 1A shows an optical sensorcoupled with optical sensor controller 158 in I/O subsystem 106. Opticalsensor(s) 164 optionally include charge-coupled device (CCD) orcomplementary metal-oxide semiconductor (CMOS) phototransistors. Opticalsensor(s) 164 receive light from the environment, projected through oneor more lens, and converts the light to data representing an image. Inconjunction with imaging module 143 (also called a camera module),optical sensor(s) 164 optionally capture still images and/or video. Insome embodiments, an optical sensor is located on the back of device100, opposite touch-sensitive display system 112 on the front of thedevice, so that the touch screen is enabled for use as a viewfinder forstill and/or video image acquisition. In some embodiments, anotheroptical sensor is located on the front of the device so that the user'simage is obtained (e.g., for selfies, for videoconferencing while theuser views the other video conference participants on the touch screen,etc.).

Device 100 optionally also includes one or more contact intensitysensors 165. FIG. 1A shows a contact intensity sensor coupled withintensity sensor controller 159 in I/O subsystem 106. Contact intensitysensor(s) 165 optionally include one or more piezoresistive straingauges, capacitive force sensors, electric force sensors, piezoelectricforce sensors, optical force sensors, capacitive touch-sensitivesurfaces, or other intensity sensors (e.g., sensors used to measure theforce (or pressure) of a contact on a touch-sensitive surface). Contactintensity sensor(s) 165 receive contact intensity information (e.g.,pressure information or a proxy for pressure information) from theenvironment. In some embodiments, at least one contact intensity sensoris collocated with, or proximate to, a touch-sensitive surface (e.g.,touch-sensitive display system 112). In some embodiments, at least onecontact intensity sensor is located on the back of device 100, oppositetouch-screen display system 112 which is located on the front of device100.

Device 100 optionally also includes one or more proximity sensors 166.FIG. 1A shows proximity sensor 166 coupled with peripherals interface118. Alternately, proximity sensor 166 is coupled with input controller160 in I/O subsystem 106. In some embodiments, the proximity sensorturns off and disables touch-sensitive display system 112 when themultifunction device is placed near the user's ear (e.g., when the useris making a phone call).

Device 100 optionally also includes one or more tactile outputgenerators 163. FIG. 1A shows a tactile output generator coupled withhaptic feedback controller 161 in I/O subsystem 106. In someembodiments, tactile output generator(s) 163 include one or moreelectroacoustic devices such as speakers or other audio componentsand/or electromechanical devices that convert energy into linear motionsuch as a motor, solenoid, electroactive polymer, piezoelectricactuator, electrostatic actuator, or other tactile output generatingcomponent (e.g., a component that converts electrical signals intotactile outputs on the device). Tactile output generator(s) 163 receivetactile feedback generation instructions from haptic feedback module 133and generates tactile outputs on device 100 that are capable of beingsensed by a user of device 100. In some embodiments, at least onetactile output generator is collocated with, or proximate to, atouch-sensitive surface (e.g., touch-sensitive display system 112) and,optionally, generates a tactile output by moving the touch-sensitivesurface vertically (e.g., in/out of a surface of device 100) orlaterally (e.g., back and forth in the same plane as a surface of device100). In some embodiments, at least one tactile output generator sensoris located on the back of device 100, opposite touch-sensitive displaysystem 112, which is located on the front of device 100.

Device 100 optionally also includes one or more accelerometers 167,gyroscopes 168, and/or magnetometers 169 (e.g., as part of an inertialmeasurement unit (IMU)) for obtaining information concerning theposition (e.g., attitude) of the device. FIG. 1A shows sensors 167, 168,and 169 coupled with peripherals interface 118. Alternately, sensors167, 168, and 169 are, optionally, coupled with an input controller 160in I/O subsystem 106. In some embodiments, information is displayed onthe touch-screen display in a portrait view or a landscape view based onan analysis of data received from the one or more accelerometers. Device100 optionally includes a GPS (or GLONASS or other global navigationsystem) receiver (not shown) for obtaining information concerning thelocation of device 100.

In some embodiments, the software components stored in memory 102include operating system 126, communication module (or set ofinstructions) 128, contact/motion module (or set of instructions) 130,graphics module (or set of instructions) 132, haptic feedback module (orset of instructions) 133, text input module (or set of instructions)134, Global Positioning System (GPS) module (or set of instructions)135, and applications (or sets of instructions) 136. Furthermore, insome embodiments, memory 102 stores device/global internal state 157, asshown in FIGS. 1A and 3. Device/global internal state 157 includes oneor more of: active application state, indicating which applications, ifany, are currently active; display state, indicating what applications,views or other information occupy various regions of touch-sensitivedisplay system 112; sensor state, including information obtained fromthe device's various sensors and other input or control devices 116; andlocation and/or positional information concerning the device's locationand/or attitude.

Operating system 126 (e.g., iOS, Android, Darwin, RTXC, LINUX, UNIX, OSX, WINDOWS, or an embedded operating system such as VxWorks) includesvarious software components and/or drivers for controlling and managinggeneral system tasks (e.g., memory management, storage device control,power management, etc.) and facilitates communication between varioushardware and software components.

Communication module 128 facilitates communication with other devicesover one or more external ports 124 and also includes various softwarecomponents for handling data received by RF circuitry 108 and/orexternal port 124. External port 124 (e.g., Universal Serial Bus (USB),FIREWIRE, etc.) is adapted for coupling directly to other devices orindirectly over a network (e.g., the Internet, wireless LAN, etc.). Insome embodiments, the external port is a multi-pin (e.g., 30-pin)connector that is the same as, or similar to and/or compatible with the30-pin connector used in some iPhone®, iPod Touch®, and iPad® devicesfrom Apple Inc. of Cupertino, Calif. In some embodiments, the externalport is a Lightning connector that is the same as, or similar to and/orcompatible with the Lightning connector used in some iPhone®, iPodTouch®, and iPad® devices from Apple Inc. of Cupertino, Calif. In someembodiments, the external port is a USB Type-C connector that is thesame as, or similar to and/or compatible with the USB Type-C connectorused in some electronic devices from Apple Inc. of Cupertino, Calif.

Contact/motion module 130 optionally detects contact withtouch-sensitive display system 112 (in conjunction with displaycontroller 156) and other touch-sensitive devices (e.g., a touchpad orphysical click wheel). Contact/motion module 130 includes varioussoftware components for performing various operations related todetection of contact (e.g., by a finger or by a stylus), such asdetermining if contact has occurred (e.g., detecting a finger-downevent), determining an intensity of the contact (e.g., the force orpressure of the contact or a substitute for the force or pressure of thecontact), determining if there is movement of the contact and trackingthe movement across the touch-sensitive surface (e.g., detecting one ormore finger-dragging events), and determining if the contact has ceased(e.g., detecting a finger-up event or a break in contact).Contact/motion module 130 receives contact data from the touch-sensitivesurface. Determining movement of the point of contact, which isrepresented by a series of contact data, optionally includes determiningspeed (magnitude), velocity (magnitude and direction), and/or anacceleration (a change in magnitude and/or direction) of the point ofcontact. These operations are, optionally, applied to single contacts(e.g., one finger contacts or stylus contacts) or to multiplesimultaneous contacts (e.g., “multitouch”/multiple finger contacts). Insome embodiments, contact/motion module 130 and display controller 156detect contact on a touchpad.

Contact/motion module 130 optionally detects a gesture input by a user.Different gestures on the touch-sensitive surface have different contactpatterns (e.g., different motions, timings, and/or intensities ofdetected contacts). Thus, a gesture is, optionally, detected bydetecting a particular contact pattern. For example, detecting a fingertap gesture includes detecting a finger-down event followed by detectinga finger-up (lift off) event at the same position (or substantially thesame position) as the finger-down event (e.g., at the position of anicon). As another example, detecting a finger swipe gesture on thetouch-sensitive surface includes detecting a finger-down event followedby detecting one or more finger-dragging events, and subsequentlyfollowed by detecting a finger-up (lift off) event. Similarly, tap,swipe, drag, and other gestures are optionally detected for a stylus bydetecting a particular contact pattern for the stylus.

In some embodiments, detecting a finger tap gesture depends on thelength of time between detecting the finger-down event and the finger-upevent, but is independent of the intensity of the finger contact betweendetecting the finger-down event and the finger-up event. In someembodiments, a tap gesture is detected in accordance with adetermination that the length of time between the finger-down event andthe finger-up event is less than a predetermined value (e.g., less than0.1, 0.2, 0.3, 0.4 or 0.5 seconds), independent of whether the intensityof the finger contact during the tap meets a given intensity threshold(greater than a nominal contact-detection intensity threshold), such asa light press or deep press intensity threshold. Thus, a finger tapgesture can satisfy particular input criteria that do not require thatthe characteristic intensity of a contact satisfy a given intensitythreshold in order for the particular input criteria to be met. Forclarity, the finger contact in a tap gesture typically needs to satisfya nominal contact-detection intensity threshold, below which the contactis not detected, in order for the finger-down event to be detected. Asimilar analysis applies to detecting a tap gesture by a stylus or othercontact. In cases where the device is capable of detecting a finger orstylus contact hovering over a touch sensitive surface, the nominalcontact-detection intensity threshold optionally does not correspond tophysical contact between the finger or stylus and the touch sensitivesurface.

The same concepts apply in an analogous manner to other types ofgestures. For example, a swipe gesture, a pinch gesture, a depinchgesture, and/or a long press gesture are optionally detected based onthe satisfaction of criteria that are either independent of intensitiesof contacts included in the gesture, or do not require that contact(s)that perform the gesture reach intensity thresholds in order to berecognized. For example, a swipe gesture is detected based on an amountof movement of one or more contacts; a pinch gesture is detected basedon movement of two or more contacts towards each other; a depinchgesture is detected based on movement of two or more contacts away fromeach other; and a long press gesture is detected based on a duration ofthe contact on the touch-sensitive surface with less than a thresholdamount of movement. As such, the statement that particular gesturerecognition criteria do not require that the intensity of the contact(s)meet a respective intensity threshold in order for the particulargesture recognition criteria to be met means that the particular gesturerecognition criteria are capable of being satisfied if the contact(s) inthe gesture do not reach the respective intensity threshold, and arealso capable of being satisfied in circumstances where one or more ofthe contacts in the gesture do reach or exceed the respective intensitythreshold. In some embodiments, a tap gesture is detected based on adetermination that the finger-down and finger-up event are detectedwithin a predefined time period, without regard to whether the contactis above or below the respective intensity threshold during thepredefined time period, and a swipe gesture is detected based on adetermination that the contact movement is greater than a predefinedmagnitude, even if the contact is above the respective intensitythreshold at the end of the contact movement. Even in implementationswhere detection of a gesture is influenced by the intensity of contactsperforming the gesture (e.g., the device detects a long press morequickly when the intensity of the contact is above an intensitythreshold or delays detection of a tap input when the intensity of thecontact is higher), the detection of those gestures does not requirethat the contacts reach a particular intensity threshold so long as thecriteria for recognizing the gesture can be met in circumstances wherethe contact does not reach the particular intensity threshold (e.g.,even if the amount of time that it takes to recognize the gesturechanges).

Contact intensity thresholds, duration thresholds, and movementthresholds are, in some circumstances, combined in a variety ofdifferent combinations in order to create heuristics for distinguishingtwo or more different gestures directed to the same input element orregion so that multiple different interactions with the same inputelement are enabled to provide a richer set of user interactions andresponses. The statement that a particular set of gesture recognitioncriteria do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the particular gesturerecognition criteria to be met does not preclude the concurrentevaluation of other intensity-dependent gesture recognition criteria toidentify other gestures that do have criteria that are met when agesture includes a contact with an intensity above the respectiveintensity threshold. For example, in some circumstances, first gesturerecognition criteria for a first gesture—which do not require that theintensity of the contact(s) meet a respective intensity threshold inorder for the first gesture recognition criteria to be met—are incompetition with second gesture recognition criteria for a secondgesture—which are dependent on the contact(s) reaching the respectiveintensity threshold. In such competitions, the gesture is, optionally,not recognized as meeting the first gesture recognition criteria for thefirst gesture if the second gesture recognition criteria for the secondgesture are met first. For example, if a contact reaches the respectiveintensity threshold before the contact moves by a predefined amount ofmovement, a deep press gesture is detected rather than a swipe gesture.Conversely, if the contact moves by the predefined amount of movementbefore the contact reaches the respective intensity threshold, a swipegesture is detected rather than a deep press gesture. Even in suchcircumstances, the first gesture recognition criteria for the firstgesture still do not require that the intensity of the contact(s) meet arespective intensity threshold in order for the first gesturerecognition criteria to be met because if the contact stayed below therespective intensity threshold until an end of the gesture (e.g., aswipe gesture with a contact that does not increase to an intensityabove the respective intensity threshold), the gesture would have beenrecognized by the first gesture recognition criteria as a swipe gesture.As such, particular gesture recognition criteria that do not requirethat the intensity of the contact(s) meet a respective intensitythreshold in order for the particular gesture recognition criteria to bemet will (A) in some circumstances ignore the intensity of the contactwith respect to the intensity threshold (e.g. for a tap gesture) and/or(B) in some circumstances still be dependent on the intensity of thecontact with respect to the intensity threshold in the sense that theparticular gesture recognition criteria (e.g., for a long press gesture)will fail if a competing set of intensity-dependent gesture recognitioncriteria (e.g., for a deep press gesture) recognize an input ascorresponding to an intensity-dependent gesture before the particulargesture recognition criteria recognize a gesture corresponding to theinput (e.g., for a long press gesture that is competing with a deeppress gesture for recognition).

Attitude module 131, in conjunction with accelerometers 167, gyroscopes168, and/or magnetometers 169, optionally detects attitude informationconcerning the device, such as the device's attitude (e.g., roll, pitch,and/or yaw) in a particular frame of reference. Attitude module 131includes software components for performing various operations relatedto detecting the position of the device and detecting changes to theattitude of the device.

Graphics module 132 includes various known software components forrendering and displaying graphics on touch-sensitive display system 112or other display, including components for changing the visual impact(e.g., brightness, transparency, saturation, contrast or other visualproperty) of graphics that are displayed. As used herein, the term“graphics” includes any object that can be displayed to a user,including without limitation text, web pages, icons (such asuser-interface objects including soft keys), digital images, videos,animations and the like.

In some embodiments, graphics module 132 stores data representinggraphics to be used. Each graphic is, optionally, assigned acorresponding code. Graphics module 132 receives, from applicationsetc., one or more codes specifying graphics to be displayed along with,if necessary, coordinate data and other graphic property data, and thengenerates screen image data to output to display controller 156.

Haptic feedback module 133 includes various software components forgenerating instructions (e.g., instructions used by haptic feedbackcontroller 161) to produce tactile outputs using tactile outputgenerator(s) 163 at one or more locations on device 100 in response touser interactions with device 100.

Text input module 134, which is, optionally, a component of graphicsmodule 132, provides soft keyboards for entering text in variousapplications (e.g., contacts 137, e-mail 140, IM 141, browser 147, andany other application that needs text input).

GPS module 135 determines the location of the device and provides thisinformation for use in various applications (e.g., to telephone 138 foruse in location-based dialing, to camera 143 as picture/video metadata,and to applications that provide location-based services such as weatherwidgets, local yellow page widgets, and map/navigation widgets).

Virtual/augmented reality module 145 provides virtual and/or augmentedreality logic to applications 136 that implement augmented reality, andin some embodiments virtual reality, features. Virtual/augmented realitymodule 145 facilitates superposition of virtual content, such as avirtual user interface object (e.g., a virtual measuring tape for makingaugmented reality-based measurements), on a representation of at least aportion of a field of view of the one or more cameras. For example, withassistance from the virtual/augmented reality module 145, therepresentation of at least a portion of a field of view of the one ormore cameras may include a respective physical object and the virtualuser interface object may be displayed at a location, in a displayedaugmented reality environment, that is determined based on therespective physical object in the field of view of the one or morecameras or a virtual reality environment that is determined based on theattitude of at least a portion of a computer system (e.g., an attitudeof a display device that is used to display the user interface to a userof the computer system).

Applications 136 optionally include the following modules (or sets ofinstructions), or a subset or superset thereof:

-   -   contacts module 137 (sometimes called an address book or contact        list);    -   telephone module 138;    -   video conferencing module 139;    -   e-mail client module 140;    -   instant messaging (IM) module 141;    -   workout support module 142;    -   camera module 143 for still and/or video images;    -   image management module 144;    -   browser module 147;    -   calendar module 148;    -   widget modules 149, which optionally include one or more of:        weather widget 149-1, stocks widget 149-2, calculator widget        149-3, alarm clock widget 149-4, dictionary widget 149-5, and        other widgets obtained by the user, as well as user-created        widgets 149-6;    -   widget creator module 150 for making user-created widgets 149-6;    -   search module 151;    -   video and music player module 152, which is, optionally, made up        of a video player module and a music player module;    -   notes module 153;    -   map module 154; and/or    -   measurement module 155.

Examples of other applications 136 that are, optionally, stored inmemory 102 include other word processing applications, other imageediting applications, drawing applications, presentation applications,JAVA-enabled applications, encryption, digital rights management, voicerecognition, and voice replication.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, contacts module 137 includes executable instructions tomanage an address book or contact list (e.g., stored in applicationinternal state 192 of contacts module 137 in memory 102 or memory 370),including: adding name(s) to the address book; deleting name(s) from theaddress book; associating telephone number(s), e-mail address(es),physical address(es) or other information with a name; associating animage with a name; categorizing and sorting names; providing telephonenumbers and/or e-mail addresses to initiate and/or facilitatecommunications by telephone 138, video conference 139, e-mail 140, or IM141; and so forth.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, contact module 130, graphics module 132, and text input module 134,telephone module 138 includes executable instructions to enter asequence of characters corresponding to a telephone number, access oneor more telephone numbers in address book 137, modify a telephone numberthat has been entered, dial a respective telephone number, conduct aconversation and disconnect or hang up when the conversation iscompleted. As noted above, the wireless communication optionally usesany of a plurality of communications standards, protocols andtechnologies.

In conjunction with RF circuitry 108, audio circuitry 110, speaker 111,microphone 113, touch-sensitive display system 112, display controller156, optical sensor(s) 164, optical sensor controller 158, contactmodule 130, graphics module 132, text input module 134, contact list137, and telephone module 138, videoconferencing module 139 includesexecutable instructions to initiate, conduct, and terminate a videoconference between a user and one or more other participants inaccordance with user instructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, e-mail client module 140 includes executableinstructions to create, send, receive, and manage e-mail in response touser instructions. In conjunction with image management module 144,e-mail client module 140 makes it very easy to create and send e-mailswith still or video images taken with camera module 143.

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,and text input module 134, the instant messaging module 141 includesexecutable instructions to enter a sequence of characters correspondingto an instant message, to modify previously entered characters, totransmit a respective instant message (for example, using a ShortMessage Service (SMS) or Multimedia Message Service (MMS) protocol fortelephony-based instant messages or using XMPP, SIMPLE, Apple PushNotification Service (APNs) or IMPS for Internet-based instantmessages), to receive instant messages, and to view received instantmessages. In some embodiments, transmitted and/or received instantmessages optionally include graphics, photos, audio files, video filesand/or other attachments as are supported in a MMS and/or an EnhancedMessaging Service (EMS). As used herein, “instant messaging” refers toboth telephony-based messages (e.g., messages sent using SMS or MMS) andInternet-based messages (e.g., messages sent using XMPP, SIMPLE, APNs,or IMPS).

In conjunction with RF circuitry 108, touch-sensitive display system112, display controller 156, contact module 130, graphics module 132,text input module 134, GPS module 135, map module 154, and video andmusic player module 152, workout support module 142 includes executableinstructions to create workouts (e.g., with time, distance, and/orcalorie burning goals); communicate with workout sensors (in sportsdevices and smart watches); receive workout sensor data; calibratesensors used to monitor a workout; select and play music for a workout;and display, store and transmit workout data.

In conjunction with touch-sensitive display system 112, displaycontroller 156, optical sensor(s) 164, optical sensor controller 158,contact module 130, graphics module 132, and image management module144, camera module 143 includes executable instructions to capture stillimages or video (including a video stream) and store them into memory102, modify characteristics of a still image or video, and/or delete astill image or video from memory 102.

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, text inputmodule 134, and camera module 143, image management module 144 includesexecutable instructions to arrange, modify (e.g., edit), or otherwisemanipulate, label, delete, present (e.g., in a digital slide show oralbum), and store still and/or video images.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, and text input module 134, browser module 147 includes executableinstructions to browse the Internet in accordance with userinstructions, including searching, linking to, receiving, and displayingweb pages or portions thereof, as well as attachments and other fileslinked to web pages.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, e-mail client module 140, and browser module147, calendar module 148 includes executable instructions to create,display, modify, and store calendars and data associated with calendars(e.g., calendar entries, to do lists, etc.) in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, widget modules 149are mini-applications that are, optionally, downloaded and used by auser (e.g., weather widget 149-1, stocks widget 149-2, calculator widget149-3, alarm clock widget 149-4, and dictionary widget 149-5) or createdby the user (e.g., user-created widget 149-6). In some embodiments, awidget includes an HTML (Hypertext Markup Language) file, a CSS(Cascading Style Sheets) file, and a JavaScript file. In someembodiments, a widget includes an XML (Extensible Markup Language) fileand a JavaScript file (e.g., Yahoo! Widgets).

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, and browser module 147, the widget creatormodule 150 includes executable instructions to create widgets (e.g.,turning a user-specified portion of a web page into a widget).

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, and text inputmodule 134, search module 151 includes executable instructions to searchfor text, music, sound, image, video, and/or other files in memory 102that match one or more search criteria (e.g., one or more user-specifiedsearch terms) in accordance with user instructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, audio circuitry110, speaker 111, RF circuitry 108, and browser module 147, video andmusic player module 152 includes executable instructions that allow theuser to download and play back recorded music and other sound filesstored in one or more file formats, such as MP3 or AAC files, andexecutable instructions to display, present or otherwise play backvideos (e.g., on touch-sensitive display system 112, or on an externaldisplay connected wirelessly or via external port 124). In someembodiments, device 100 optionally includes the functionality of an MP3player, such as an iPod (trademark of Apple Inc.).

In conjunction with touch-sensitive display system 112, displaycontroller 156, contact module 130, graphics module 132, and text inputmodule 134, notes module 153 includes executable instructions to createand manage notes, to do lists, and the like in accordance with userinstructions.

In conjunction with RF circuitry 108, touch-sensitive display system112, display system controller 156, contact module 130, graphics module132, text input module 134, GPS module 135, and browser module 147, mapmodule 154 includes executable instructions to receive, display, modify,and store maps and data associated with maps (e.g., driving directions;data on stores and other points of interest at or near a particularlocation; and other location-based data) in accordance with userinstructions.

In conjunction with touch-sensitive display system 112, display systemcontroller 156, contact module 130, graphics module 132, andvirtual/augmented reality module 145, measurement module 155 includesexecutable instructions that allow the user to measure physical spacesand/or objects therein in an augmented reality environment, as describedin more detail herein.

Each of the above identified modules and applications correspond to aset of executable instructions for performing one or more functionsdescribed above and the methods described in this application (e.g., thecomputer-implemented methods and other information processing methodsdescribed herein). These modules (i.e., sets of instructions) need notbe implemented as separate software programs, procedures or modules, andthus various subsets of these modules are, optionally, combined orotherwise re-arranged in various embodiments. In some embodiments,memory 102 optionally stores a subset of the modules and data structuresidentified above. Furthermore, memory 102 optionally stores additionalmodules and data structures not described above.

In some embodiments, device 100 is a device where operation of apredefined set of functions on the device is performed exclusivelythrough a touch screen and/or a touchpad. By using a touch screen and/ora touchpad as the primary input control device for operation of device100, the number of physical input control devices (such as push buttons,dials, and the like) on device 100 is, optionally, reduced.

The predefined set of functions that are performed exclusively through atouch screen and/or a touchpad optionally include navigation betweenuser interfaces. In some embodiments, the touchpad, when touched by theuser, navigates device 100 to a main, home, or root menu from any userinterface that is displayed on device 100. In such embodiments, a “menubutton” is implemented using a touch-sensitive surface. In some otherembodiments, the menu button is a physical push button or other physicalinput control device instead of a touch-sensitive surface.

FIG. 1B is a block diagram illustrating example components for eventhandling in accordance with some embodiments. In some embodiments,memory 102 (in FIG. 1A) or 370 (FIG. 3A) includes event sorter 170(e.g., in operating system 126) and a respective application 136-1(e.g., any of the aforementioned applications 136, 137-155, 380-390).

Event sorter 170 receives event information and determines theapplication 136-1 and application view 191 of application 136-1 to whichto deliver the event information. Event sorter 170 includes eventmonitor 171 and event dispatcher module 174. In some embodiments,application 136-1 includes application internal state 192, whichindicates the current application view(s) displayed on touch-sensitivedisplay system 112 when the application is active or executing. In someembodiments, device/global internal state 157 is used by event sorter170 to determine which application(s) is (are) currently active, andapplication internal state 192 is used by event sorter 170 to determineapplication views 191 to which to deliver event information.

In some embodiments, application internal state 192 includes additionalinformation, such as one or more of: resume information to be used whenapplication 136-1 resumes execution, user interface state informationthat indicates information being displayed or that is ready for displayby application 136-1, a state queue for enabling the user to go back toa prior state or view of application 136-1, and a redo/undo queue ofprevious actions taken by the user.

Event monitor 171 receives event information from peripherals interface118. Event information includes information about a sub-event (e.g., auser touch on touch-sensitive display system 112, as part of amulti-touch gesture). Peripherals interface 118 transmits information itreceives from I/O subsystem 106 or a sensor, such as proximity sensor166, accelerometer(s) 167, and/or microphone 113 (through audiocircuitry 110). Information that peripherals interface 118 receives fromI/O subsystem 106 includes information from touch-sensitive displaysystem 112 or a touch-sensitive surface.

In some embodiments, event monitor 171 sends requests to the peripheralsinterface 118 at predetermined intervals. In response, peripheralsinterface 118 transmits event information. In other embodiments,peripheral interface 118 transmits event information only when there isa significant event (e.g., receiving an input above a predeterminednoise threshold and/or for more than a predetermined duration).

In some embodiments, event sorter 170 also includes a hit viewdetermination module 172 and/or an active event recognizer determinationmodule 173.

Hit view determination module 172 provides software procedures fordetermining where a sub-event has taken place within one or more views,when touch-sensitive display system 112 displays more than one view.Views are made up of controls and other elements that a user can see onthe display.

Another aspect of the user interface associated with an application is aset of views, sometimes herein called application views or userinterface windows, in which information is displayed and touch-basedgestures occur. The application views (of a respective application) inwhich a touch is detected optionally correspond to programmatic levelswithin a programmatic or view hierarchy of the application. For example,the lowest level view in which a touch is detected is, optionally,called the hit view, and the set of events that are recognized as properinputs are, optionally, determined based, at least in part, on the hitview of the initial touch that begins a touch-based gesture.

Hit view determination module 172 receives information related tosub-events of a touch-based gesture. When an application has multipleviews organized in a hierarchy, hit view determination module 172identifies a hit view as the lowest view in the hierarchy which shouldhandle the sub-event. In most circumstances, the hit view is the lowestlevel view in which an initiating sub-event occurs (i.e., the firstsub-event in the sequence of sub-events that form an event or potentialevent). Once the hit view is identified by the hit view determinationmodule, the hit view typically receives all sub-events related to thesame touch or input source for which it was identified as the hit view.

Active event recognizer determination module 173 determines which viewor views within a view hierarchy should receive a particular sequence ofsub-events. In some embodiments, active event recognizer determinationmodule 173 determines that only the hit view should receive a particularsequence of sub-events. In other embodiments, active event recognizerdetermination module 173 determines that all views that include thephysical location of a sub-event are actively involved views, andtherefore determines that all actively involved views should receive aparticular sequence of sub-events. In other embodiments, even if touchsub-events were entirely confined to the area associated with oneparticular view, views higher in the hierarchy would still remain asactively involved views.

Event dispatcher module 174 dispatches the event information to an eventrecognizer (e.g., event recognizer 180). In embodiments including activeevent recognizer determination module 173, event dispatcher module 174delivers the event information to an event recognizer determined byactive event recognizer determination module 173. In some embodiments,event dispatcher module 174 stores in an event queue the eventinformation, which is retrieved by a respective event receiver module182.

In some embodiments, operating system 126 includes event sorter 170.Alternatively, application 136-1 includes event sorter 170. In yet otherembodiments, event sorter 170 is a stand-alone module, or a part ofanother module stored in memory 102, such as contact/motion module 130.

In some embodiments, application 136-1 includes a plurality of eventhandlers 190 and one or more application views 191, each of whichincludes instructions for handling touch events that occur within arespective view of the application's user interface. Each applicationview 191 of the application 136-1 includes one or more event recognizers180. Typically, a respective application view 191 includes a pluralityof event recognizers 180. In other embodiments, one or more of eventrecognizers 180 are part of a separate module, such as a user interfacekit (not shown) or a higher level object from which application 136-1inherits methods and other properties. In some embodiments, a respectiveevent handler 190 includes one or more of: data updater 176, objectupdater 177, GUI updater 178, and/or event data 179 received from eventsorter 170. Event handler 190 optionally utilizes or calls data updater176, object updater 177 or GUI updater 178 to update the applicationinternal state 192. Alternatively, one or more of the application views191 includes one or more respective event handlers 190. Also, in someembodiments, one or more of data updater 176, object updater 177, andGUI updater 178 are included in a respective application view 191.

A respective event recognizer 180 receives event information (e.g.,event data 179) from event sorter 170, and identifies an event from theevent information. Event recognizer 180 includes event receiver 182 andevent comparator 184. In some embodiments, event recognizer 180 alsoincludes at least a subset of: metadata 183, and event deliveryinstructions 188 (which optionally include sub-event deliveryinstructions).

Event receiver 182 receives event information from event sorter 170. Theevent information includes information about a sub-event, for example, atouch or a touch movement. Depending on the sub-event, the eventinformation also includes additional information, such as location ofthe sub-event. When the sub-event concerns motion of a touch, the eventinformation optionally also includes speed and direction of thesub-event. In some embodiments, events include rotation of the devicefrom one orientation to another (e.g., from a portrait orientation to alandscape orientation, or vice versa), and the event informationincludes corresponding information about the current orientation (alsocalled device attitude) of the device.

Event comparator 184 compares the event information to predefined eventor sub-event definitions and, based on the comparison, determines anevent or sub-event, or determines or updates the state of an event orsub-event. In some embodiments, event comparator 184 includes eventdefinitions 186. Event definitions 186 contain definitions of events(e.g., predefined sequences of sub-events), for example, event 1(187-1), event 2 (187-2), and others. In some embodiments, sub-events inan event 187 include, for example, touch begin, touch end, touchmovement, touch cancellation, and multiple touching. In one example, thedefinition for event 1 (187-1) is a double tap on a displayed object.The double tap, for example, comprises a first touch (touch begin) onthe displayed object for a predetermined phase, a first lift-off (touchend) for a predetermined phase, a second touch (touch begin) on thedisplayed object for a predetermined phase, and a second lift-off (touchend) for a predetermined phase. In another example, the definition forevent 2 (187-2) is a dragging on a displayed object. The dragging, forexample, comprises a touch (or contact) on the displayed object for apredetermined phase, a movement of the touch across touch-sensitivedisplay system 112, and lift-off of the touch (touch end). In someembodiments, the event also includes information for one or moreassociated event handlers 190.

In some embodiments, event definition 187 includes a definition of anevent for a respective user-interface object. In some embodiments, eventcomparator 184 performs a hit test to determine which user-interfaceobject is associated with a sub-event. For example, in an applicationview in which three user-interface objects are displayed ontouch-sensitive display system 112, when a touch is detected ontouch-sensitive display system 112, event comparator 184 performs a hittest to determine which of the three user-interface objects isassociated with the touch (sub-event). If each displayed object isassociated with a respective event handler 190, the event comparatoruses the result of the hit test to determine which event handler 190should be activated. For example, event comparator 184 selects an eventhandler associated with the sub-event and the object triggering the hittest.

In some embodiments, the definition for a respective event 187 alsoincludes delayed actions that delay delivery of the event informationuntil after it has been determined whether the sequence of sub-eventsdoes or does not correspond to the event recognizer's event type.

When a respective event recognizer 180 determines that the series ofsub-events do not match any of the events in event definitions 186, therespective event recognizer 180 enters an event impossible, eventfailed, or event ended state, after which it disregards subsequentsub-events of the touch-based gesture. In this situation, other eventrecognizers, if any, that remain active for the hit view continue totrack and process sub-events of an ongoing touch-based gesture.

In some embodiments, a respective event recognizer 180 includes metadata183 with configurable properties, flags, and/or lists that indicate howthe event delivery system should perform sub-event delivery to activelyinvolved event recognizers. In some embodiments, metadata 183 includesconfigurable properties, flags, and/or lists that indicate how eventrecognizers interact, or are enabled to interact, with one another. Insome embodiments, metadata 183 includes configurable properties, flags,and/or lists that indicate whether sub-events are delivered to varyinglevels in the view or programmatic hierarchy.

In some embodiments, a respective event recognizer 180 activates eventhandler 190 associated with an event when one or more particularsub-events of an event are recognized. In some embodiments, a respectiveevent recognizer 180 delivers event information associated with theevent to event handler 190. Activating an event handler 190 is distinctfrom sending (and deferred sending) sub-events to a respective hit view.In some embodiments, event recognizer 180 throws a flag associated withthe recognized event, and event handler 190 associated with the flagcatches the flag and performs a predefined process.

In some embodiments, event delivery instructions 188 include sub-eventdelivery instructions that deliver event information about a sub-eventwithout activating an event handler. Instead, the sub-event deliveryinstructions deliver event information to event handlers associated withthe series of sub-events or to actively involved views. Event handlersassociated with the series of sub-events or with actively involved viewsreceive the event information and perform a predetermined process.

In some embodiments, data updater 176 creates and updates data used inapplication 136-1. For example, data updater 176 updates the telephonenumber used in contacts module 137, or stores a video file used in videoand music player module 152. In some embodiments, object updater 177creates and updates objects used in application 136-1. For example,object updater 177 creates a new user-interface object or updates theposition of a user-interface object. GUI updater 178 updates the GUI.For example, GUI updater 178 prepares display information and sends itto graphics module 132 for display on a touch-sensitive display.

In some embodiments, event handler(s) 190 includes or has access to dataupdater 176, object updater 177, and GUI updater 178. In someembodiments, data updater 176, object updater 177, and GUI updater 178are included in a single module of a respective application 136-1 orapplication view 191. In other embodiments, they are included in two ormore software modules.

It shall be understood that the foregoing discussion regarding eventhandling of user touches on touch-sensitive displays also applies toother forms of user inputs to operate multifunction devices 100 withinput-devices, not all of which are initiated on touch screens. Forexample, mouse movement and mouse button presses, optionally coordinatedwith single or multiple keyboard presses or holds; contact movementssuch as taps, drags, scrolls, etc., on touch-pads; pen stylus inputs;inputs based on real-time analysis of video images obtained by one ormore cameras; movement of the device; oral instructions; detected eyemovements; biometric inputs; and/or any combination thereof areoptionally utilized as inputs corresponding to sub-events which definean event to be recognized.

FIG. 1C is a block diagram illustrating a tactile output module inaccordance with some embodiments. In some embodiments, I/O subsystem 106(e.g., haptic feedback controller 161 (FIG. 1A) and/or other inputcontroller(s) 160 (FIG. 1A)) includes at least some of the examplecomponents shown in FIG. 1C. In some embodiments, peripherals interface118 includes at least some of the example components shown in FIG. 1C.

In some embodiments, the tactile output module includes haptic feedbackmodule 133. In some embodiments, haptic feedback module 133 aggregatesand combines tactile outputs for user interface feedback from softwareapplications on the electronic device (e.g., feedback that is responsiveto user inputs that correspond to displayed user interfaces and alertsand other notifications that indicate the performance of operations oroccurrence of events in user interfaces of the electronic device).Haptic feedback module 133 includes one or more of: waveform module 123(for providing waveforms used for generating tactile outputs), mixer 125(for mixing waveforms, such as waveforms in different channels),compressor 127 (for reducing or compressing a dynamic range of thewaveforms), low-pass filter 129 (for filtering out high frequency signalcomponents in the waveforms), and thermal controller 181 (for adjustingthe waveforms in accordance with thermal conditions). In someembodiments, haptic feedback module 133 is included in haptic feedbackcontroller 161 (FIG. 1A). In some embodiments, a separate unit of hapticfeedback module 133 (or a separate implementation of haptic feedbackmodule 133) is also included in an audio controller (e.g., audiocircuitry 110, FIG. 1A) and used for generating audio signals. In someembodiments, a single haptic feedback module 133 is used for generatingaudio signals and generating waveforms for tactile outputs.

In some embodiments, haptic feedback module 133 also includes triggermodule 121 (e.g., a software application, operating system, or othersoftware module that determines a tactile output is to be generated andinitiates the process for generating the corresponding tactile output).In some embodiments, trigger module 121 generates trigger signals forinitiating generation of waveforms (e.g., by waveform module 123). Forexample, trigger module 121 generates trigger signals based on presettiming criteria. In some embodiments, trigger module 121 receivestrigger signals from outside haptic feedback module 133 (e.g., in someembodiments, haptic feedback module 133 receives trigger signals fromhardware input processing module 146 located outside haptic feedbackmodule 133) and relays the trigger signals to other components withinhaptic feedback module 133 (e.g., waveform module 123) or softwareapplications that trigger operations (e.g., with trigger module 121)based on activation of a user interface element (e.g., an applicationicon or an affordance within an application) or a hardware input device(e.g., a home button or an intensity-sensitive input surface, such as anintensity-sensitive touch screen). In some embodiments, trigger module121 also receives tactile feedback generation instructions (e.g., fromhaptic feedback module 133, FIGS. 1A and 3). In some embodiments,trigger module 121 generates trigger signals in response to hapticfeedback module 133 (or trigger module 121 in haptic feedback module133) receiving tactile feedback instructions (e.g., from haptic feedbackmodule 133, FIGS. 1A and 3).

Waveform module 123 receives trigger signals (e.g., from trigger module121) as an input, and in response to receiving trigger signals, provideswaveforms for generation of one or more tactile outputs (e.g., waveformsselected from a predefined set of waveforms designated for use bywaveform module 123, such as the waveforms described in greater detailbelow with reference to FIGS. 4F-4G).

Mixer 125 receives waveforms (e.g., from waveform module 123) as aninput, and mixes together the waveforms. For example, when mixer 125receives two or more waveforms (e.g., a first waveform in a firstchannel and a second waveform that at least partially overlaps with thefirst waveform in a second channel) mixer 125 outputs a combinedwaveform that corresponds to a sum of the two or more waveforms. In someembodiments, mixer 125 also modifies one or more waveforms of the two ormore waveforms to emphasize particular waveform(s) over the rest of thetwo or more waveforms (e.g., by increasing a scale of the particularwaveform(s) and/or decreasing a scale of the rest of the waveforms). Insome circumstances, mixer 125 selects one or more waveforms to removefrom the combined waveform (e.g., the waveform from the oldest source isdropped when there are waveforms from more than three sources that havebeen requested to be output concurrently by tactile output generator163).

Compressor 127 receives waveforms (e.g., a combined waveform from mixer125) as an input, and modifies the waveforms. In some embodiments,compressor 127 reduces the waveforms (e.g., in accordance with physicalspecifications of tactile output generators 163 (FIG. 1A) or 357 (FIG.3)) so that tactile outputs corresponding to the waveforms are reduced.In some embodiments, compressor 127 limits the waveforms, such as byenforcing a predefined maximum amplitude for the waveforms. For example,compressor 127 reduces amplitudes of portions of waveforms that exceed apredefined amplitude threshold while maintaining amplitudes of portionsof waveforms that do not exceed the predefined amplitude threshold. Insome embodiments, compressor 127 reduces a dynamic range of thewaveforms. In some embodiments, compressor 127 dynamically reduces thedynamic range of the waveforms so that the combined waveforms remainwithin performance specifications of the tactile output generator 163(e.g., force and/or moveable mass displacement limits).

Low-pass filter 129 receives waveforms (e.g., compressed waveforms fromcompressor 127) as an input, and filters (e.g., smooths) the waveforms(e.g., removes or reduces high frequency signal components in thewaveforms). For example, in some instances, compressor 127 includes, incompressed waveforms, extraneous signals (e.g., high frequency signalcomponents) that interfere with the generation of tactile outputs and/orexceed performance specifications of tactile output generator 163 whenthe tactile outputs are generated in accordance with the compressedwaveforms. Low-pass filter 129 reduces or removes such extraneoussignals in the waveforms.

Thermal controller 181 receives waveforms (e.g., filtered waveforms fromlow-pass filter 129) as an input, and adjusts the waveforms inaccordance with thermal conditions of device 100 (e.g., based oninternal temperatures detected within device 100, such as thetemperature of haptic feedback controller 161, and/or externaltemperatures detected by device 100). For example, in some cases, theoutput of haptic feedback controller 161 varies depending on thetemperature (e.g. haptic feedback controller 161, in response toreceiving same waveforms, generates a first tactile output when hapticfeedback controller 161 is at a first temperature and generates a secondtactile output when haptic feedback controller 161 is at a secondtemperature that is distinct from the first temperature). For example,the magnitude (or the amplitude) of the tactile outputs may varydepending on the temperature. To reduce the effect of the temperaturevariations, the waveforms are modified (e.g., an amplitude of thewaveforms is increased or decreased based on the temperature).

In some embodiments, haptic feedback module 133 (e.g., trigger module121) is coupled to hardware input processing module 146. In someembodiments, other input controller(s) 160 in FIG. 1A includes hardwareinput processing module 146. In some embodiments, hardware inputprocessing module 146 receives inputs from hardware input device 175(e.g., other input or control devices 116 in FIG. 1A, such as a homebutton or an intensity-sensitive input surface, such as anintensity-sensitive touch screen). In some embodiments, hardware inputdevice 175 is any input device described herein, such as touch-sensitivedisplay system 112 (FIG. 1A), keyboard/mouse 350 (FIG. 3), touchpad 355(FIG. 3), one of other input or control devices 116 (FIG. 1A), or anintensity-sensitive home button. In some embodiments, hardware inputdevice 175 consists of an intensity-sensitive home button, and nottouch-sensitive display system 112 (FIG. 1A), keyboard/mouse 350 (FIG.3), or touchpad 355 (FIG. 3). In some embodiments, in response to inputsfrom hardware input device 175 (e.g., an intensity-sensitive home buttonor a touch screen), hardware input processing module 146 provides one ormore trigger signals to haptic feedback module 133 to indicate that auser input satisfying predefined input criteria, such as an inputcorresponding to a “click” of a home button (e.g., a “down click” or an“up click”), has been detected. In some embodiments, haptic feedbackmodule 133 provides waveforms that correspond to the “click” of a homebutton in response to the input corresponding to the “click” of a homebutton, simulating a haptic feedback of pressing a physical home button.

In some embodiments, the tactile output module includes haptic feedbackcontroller 161 (e.g., haptic feedback controller 161 in FIG. 1A), whichcontrols the generation of tactile outputs. In some embodiments, hapticfeedback controller 161 is coupled to a plurality of tactile outputgenerators, and selects one or more tactile output generators of theplurality of tactile output generators and sends waveforms to theselected one or more tactile output generators for generating tactileoutputs. In some embodiments, haptic feedback controller 161 coordinatestactile output requests that correspond to activation of hardware inputdevice 175 and tactile output requests that correspond to softwareevents (e.g., tactile output requests from haptic feedback module 133)and modifies one or more waveforms of the two or more waveforms toemphasize particular waveform(s) over the rest of the two or morewaveforms (e.g., by increasing a scale of the particular waveform(s)and/or decreasing a scale of the rest of the waveforms, such as toprioritize tactile outputs that correspond to activations of hardwareinput device 175 over tactile outputs that correspond to softwareevents).

In some embodiments, as shown in FIG. 1C, an output of haptic feedbackcontroller 161 is coupled to audio circuitry of device 100 (e.g., audiocircuitry 110, FIG. 1A), and provides audio signals to audio circuitryof device 100. In some embodiments, haptic feedback controller 161provides both waveforms used for generating tactile outputs and audiosignals used for providing audio outputs in conjunction with generationof the tactile outputs. In some embodiments, haptic feedback controller161 modifies audio signals and/or waveforms (used for generating tactileoutputs) so that the audio outputs and the tactile outputs aresynchronized (e.g., by delaying the audio signals and/or waveforms). Insome embodiments, haptic feedback controller 161 includes adigital-to-analog converter used for converting digital waveforms intoanalog signals, which are received by amplifier 185 and/or tactileoutput generator 163.

In some embodiments, the tactile output module includes amplifier 185.In some embodiments, amplifier 185 receives waveforms (e.g., from hapticfeedback controller 161) and amplifies the waveforms prior to sendingthe amplified waveforms to tactile output generator 163 (e.g., any oftactile output generators 163 (FIG. 1A) or 357 (FIG. 3)). For example,amplifier 185 amplifies the received waveforms to signal levels that arein accordance with physical specifications of tactile output generator163 (e.g., to a voltage and/or a current required by tactile outputgenerator 163 for generating tactile outputs so that the signals sent totactile output generator 163 produce tactile outputs that correspond tothe waveforms received from haptic feedback controller 161) and sendsthe amplified waveforms to tactile output generator 163. In response,tactile output generator 163 generates tactile outputs (e.g., byshifting a moveable mass back and forth in one or more dimensionsrelative to a neutral position of the moveable mass).

In some embodiments, the tactile output module includes sensor 189,which is coupled to tactile output generator 163. Sensor 189 detectsstates or state changes (e.g., mechanical position, physicaldisplacement, and/or movement) of tactile output generator 163 or one ormore components of tactile output generator 163 (e.g., one or moremoving parts, such as a membrane, used to generate tactile outputs). Insome embodiments, sensor 189 is a magnetic field sensor (e.g., a Halleffect sensor) or other displacement and/or movement sensor. In someembodiments, sensor 189 provides information (e.g., a position, adisplacement, and/or a movement of one or more parts in tactile outputgenerator 163) to haptic feedback controller 161 and, in accordance withthe information provided by sensor 189 about the state of tactile outputgenerator 163, haptic feedback controller 161 adjusts the waveformsoutput from haptic feedback controller 161 (e.g., waveforms sent totactile output generator 163, optionally via amplifier 185).

FIG. 2 illustrates a portable multifunction device 100 having a touchscreen (e.g., touch-sensitive display system 112, FIG. 1A) in accordancewith some embodiments. The touch screen optionally displays one or moregraphics within user interface (UI) 200. In these embodiments, as wellas others described below, a user is enabled to select one or more ofthe graphics by making a gesture on the graphics, for example, with oneor more fingers 202 (not drawn to scale in the figure) or one or morestyluses 203 (not drawn to scale in the figure). In some embodiments,selection of one or more graphics occurs when the user breaks contactwith the one or more graphics. In some embodiments, the gestureoptionally includes one or more taps, one or more swipes (from left toright, right to left, upward and/or downward) and/or a rolling of afinger (from right to left, left to right, upward and/or downward) thathas made contact with device 100. In some implementations orcircumstances, inadvertent contact with a graphic does not select thegraphic. For example, a swipe gesture that sweeps over an applicationicon optionally does not select the corresponding application when thegesture corresponding to selection is a tap.

Device 100 optionally also includes one or more physical buttons, suchas “home” or menu button 204. As described previously, menu button 204is, optionally, used to navigate to any application 136 in a set ofapplications that are, optionally executed on device 100. Alternatively,in some embodiments, the menu button is implemented as a soft key in aGUI displayed on the touch-screen display.

In some embodiments, device 100 includes the touch-screen display, menubutton 204 (sometimes called home button 204), push button 206 forpowering the device on/off and locking the device, volume adjustmentbutton(s) 208, Subscriber Identity Module (SIM) card slot 210, head setjack 212, and docking/charging external port 124. Push button 206 is,optionally, used to turn the power on/off on the device by depressingthe button and holding the button in the depressed state for apredefined time interval; to lock the device by depressing the buttonand releasing the button before the predefined time interval haselapsed; and/or to unlock the device or initiate an unlock process. Insome embodiments, device 100 also accepts verbal input for activation ordeactivation of some functions through microphone 113. Device 100 also,optionally, includes one or more contact intensity sensors 165 fordetecting intensities of contacts on touch-sensitive display system 112and/or one or more tactile output generators 163 for generating tactileoutputs for a user of device 100.

FIG. 3A is a block diagram of an example multifunction device with adisplay and a touch-sensitive surface in accordance with someembodiments. Device 300 need not be portable. In some embodiments,device 300 is a laptop computer, a desktop computer, a tablet computer,a multimedia player device, a navigation device, an educational device(such as a child's learning toy), a gaming system, or a control device(e.g., a home or industrial controller). Device 300 typically includesone or more processing units (CPU's) 310, one or more network or othercommunications interfaces 360, memory 370, and one or more communicationbuses 320 for interconnecting these components. Communication buses 320optionally include circuitry (sometimes called a chipset) thatinterconnects and controls communications between system components.Device 300 includes input/output (I/O) interface 330 comprising display340, which is optionally a touch-screen display. I/O interface 330 alsooptionally includes a keyboard and/or mouse (or other pointing device)350 and touchpad 355, tactile output generator 357 for generatingtactile outputs on device 300 (e.g., similar to tactile outputgenerator(s) 163 described above with reference to FIG. 1A), sensors 359(e.g., optical, acceleration, proximity, touch-sensitive, and/or contactintensity sensors similar to contact intensity sensor(s) 165 describedabove with reference to FIG. 1A). Memory 370 includes high-speed randomaccess memory, such as DRAM, SRAM, DDR RAM or other random access solidstate memory devices; and optionally includes non-volatile memory, suchas one or more magnetic disk storage devices, optical disk storagedevices, flash memory devices, or other non-volatile solid state storagedevices. Memory 370 optionally includes one or more storage devicesremotely located from CPU(s) 310. In some embodiments, memory 370 storesprograms, modules, and data structures analogous to the programs,modules, and data structures stored in memory 102 of portablemultifunction device 100 (FIG. 1A), or a subset thereof. Furthermore,memory 370 optionally stores additional programs, modules, and datastructures not present in memory 102 of portable multifunction device100. For example, memory 370 of device 300 optionally stores drawingmodule 380, presentation module 382, word processing module 384, websitecreation module 386, disk authoring module 388, and/or spreadsheetmodule 390, while memory 102 of portable multifunction device 100 (FIG.1A) optionally does not store these modules.

Each of the above identified elements in FIG. 3A are, optionally, storedin one or more of the previously mentioned memory devices. Each of theabove identified modules corresponds to a set of instructions forperforming a function described above. The above identified modules orprograms (e.g., sets of instructions) need not be implemented asseparate software programs, procedures or modules, and thus varioussubsets of these modules are, optionally, combined or otherwisere-arranged in various embodiments. In some embodiments, memory 370optionally stores a subset of the modules and data structures identifiedabove. Furthermore, memory 370 optionally stores additional modules anddata structures not described above.

FIGS. 3B-3C are block diagrams of example computer systems 301 inaccordance with some embodiments.

In some embodiments, computer system 301 includes and/or is incommunication with:

-   -   input device(s) (302 and/or 307, e.g., a touch-sensitive        surface, such as a touch-sensitive remote control, or a        touch-screen display that also serves as the display generation        component, a mouse, a joystick, a wand controller, and/or        cameras tracking the position of one or more features of the        user such as the user's hands);    -   virtual/augmented reality logic 303 (e.g., virtual/augmented        reality module 145);    -   display generation component(s) (304 and/or 308, e.g., a        display, a projector, a heads-up display, or the like) for        displaying virtual user interface elements to the user;    -   camera(s) (e.g., 305 and/or 311) for capturing images of a field        of view of the device, e.g., images that are used to determine        placement of virtual user interface elements, determine an        attitude of the device, and/or display a portion of the physical        environment in which the camera(s) are located; and    -   attitude sensor(s) (e.g., 306 and/or 311) for determining an        attitude of the device relative to the physical environment        and/or changes in attitude of the device.

In some computer systems (e.g., 301-a in FIG. 3B), input device(s) 302,virtual/augmented reality logic 303, display generation component(s)304, camera(s) 305; and attitude sensor(s) 306 are all integrated intothe computer system (e.g., portable multifunction device 100 in FIGS.1A-1B or device 300 in FIG. 3 such as a smartphone or tablet).

In some computer systems (e.g., 301-b), in addition to integrated inputdevice(s) 302, virtual/augmented reality logic 303, display generationcomponent(s) 304, camera(s) 305; and attitude sensor(s) 306, thecomputer system is also in communication with additional devices thatare separate from the computer system, such as separate input device(s)307 such as a touch-sensitive surface, a wand, a remote control, or thelike and/or separate display generation component(s) 308 such as virtualreality headset or augmented reality glasses that overlay virtualobjects on a physical environment.

In some computer systems (e.g., 301-c in FIG. 3C), the input device(s)307, display generation component(s) 309, camera(s) 311; and/or attitudesensor(s) 312 are separate from the computer system and are incommunication with the computer system. In some embodiments, othercombinations of components in computer system 301 and in communicationwith the computer system are used. For example, in some embodiments,display generation component(s) 309, camera(s) 311, and attitudesensor(s) 312 are incorporated in a headset that is either integratedwith or in communication with the computer system.

In some embodiments, all of the operations described below withreference to FIGS. 5A-5CO are performed on a single computing devicewith virtual/augmented reality logic 303 (e.g., computer system 301-adescribed below with reference to FIG. 3B). However, it should beunderstood that frequently multiple different computing devices arelinked together to perform the operations described below with referenceto FIGS. 5A-5CO (e.g., a computing device with virtual/augmented realitylogic 303 communicates with a separate computing device with a display450 and/or a separate computing device with a touch-sensitive surface451). In any of these embodiments, the computing device that isdescribed below with reference to FIGS. 5A-5CO is the computing device(or devices) that contain(s) the virtual/augmented reality logic 303.Additionally, it should be understood that the virtual/augmented realitylogic 303 could be divided between a plurality of distinct modules orcomputing devices in various embodiments; however, for the purposes ofthe description herein, the virtual/augmented reality logic 303 will beprimarily referred to as residing in a single computing device so as notto unnecessarily obscure other aspects of the embodiments.

In some embodiments, the virtual/augmented reality logic 303 includesone or more modules (e.g., one or more event handlers 190, including oneor more object updaters 177 and one or more GUI updaters 178 asdescribed in greater detail above with reference to FIG. 1B) thatreceive interpreted inputs and, in response to these interpreted inputs,generate instructions for updating a graphical user interface inaccordance with the interpreted inputs which are subsequently used toupdate the graphical user interface on a display. In some embodiments,an interpreted input for an input that has been detected (e.g., by acontact motion module 130 in FIGS. 1A and 3), recognized (e.g., by anevent recognizer 180 in FIG. 1B) and/or distributed (e.g., by eventsorter 170 in FIG. 1B) is used to update the graphical user interface ona display. In some embodiments, the interpreted inputs are generated bymodules at the computing device (e.g., the computing device receives rawcontact input data so as to identify gestures from the raw contact inputdata). In some embodiments, some or all of the interpreted inputs arereceived by the computing device as interpreted inputs (e.g., acomputing device that includes the touch-sensitive surface 451 processesraw contact input data so as to identify gestures from the raw contactinput data and sends information indicative of the gestures to thecomputing device that includes the virtual/augmented reality logic 303).

In some embodiments, both a display and a touch-sensitive surface areintegrated with the computer system (e.g., 301-a in FIG. 3B) thatcontains the virtual/augmented reality logic 303. For example, thecomputer system may be a desktop computer or laptop computer with anintegrated display (e.g., 340 in FIG. 3) and touchpad (e.g., 355 in FIG.3). As another example, the computing device may be a portablemultifunction device 100 (e.g., a smartphone, PDA, tablet computer,etc.) with a touch screen (e.g., 112 in FIG. 2).

In some embodiments, a touch-sensitive surface is integrated with thecomputer system while a display is not integrated with the computersystem that contains the virtual/augmented reality logic 303. Forexample, the computer system may be a device 300 (e.g., a desktopcomputer or laptop computer) with an integrated touchpad (e.g., 355 inFIG. 3) connected (via wired or wireless connection) to a separatedisplay (e.g., a computer monitor, television, etc.). As anotherexample, the computer system may be a portable multifunction device 100(e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen(e.g., 112 in FIG. 2) connected (via wired or wireless connection) to aseparate display (e.g., a computer monitor, television, etc.).

In some embodiments, a display is integrated with the computer systemwhile a touch-sensitive surface is not integrated with the computersystem that contains the virtual/augmented reality logic 303. Forexample, the computer system may be a device 300 (e.g., a desktopcomputer, laptop computer, television with integrated set-top box) withan integrated display (e.g., 340 in FIG. 3) connected (via wired orwireless connection) to a separate touch-sensitive surface (e.g., aremote touchpad, a portable multifunction device, etc.). As anotherexample, the computer system may be a portable multifunction device 100(e.g., a smartphone, PDA, tablet computer, etc.) with a touch screen(e.g., 112 in FIG. 2) connected (via wired or wireless connection) to aseparate touch-sensitive surface (e.g., a remote touchpad, anotherportable multifunction device with a touch screen serving as a remotetouchpad, etc.).

In some embodiments, neither a display nor a touch-sensitive surface isintegrated with the computer system (e.g., 301-c in FIG. 3C) thatcontains the virtual/augmented reality logic 303. For example, thecomputer system may be a stand-alone computing device 300 (e.g., aset-top box, gaming console, etc.) connected (via wired or wirelessconnection) to a separate touch-sensitive surface (e.g., a remotetouchpad, a portable multifunction device, etc.) and a separate display(e.g., a computer monitor, television, etc.).

In some embodiments, the computer system has an integrated audio system(e.g., audio circuitry 110 and speaker 111 in portable multifunctiondevice 100). In some embodiments, the computing device is incommunication with an audio system that is separate from the computingdevice. In some embodiments, the audio system (e.g., an audio systemintegrated in a television unit) is integrated with a separate display.In some embodiments, the audio system (e.g., a stereo system) is astand-alone system that is separate from the computer system and thedisplay.

Attention is now directed towards embodiments of user interfaces (“UI”)that are, optionally, implemented on portable multifunction device 100.

FIG. 4A illustrates an example user interface for a menu of applicationson portable multifunction device 100 in accordance with someembodiments. Similar user interfaces are, optionally, implemented ondevice 300. In some embodiments, user interface 400 includes thefollowing elements, or a subset or superset thereof:

-   -   Signal strength indicator(s) for wireless communication(s), such        as cellular and Wi-Fi signals;    -   Time;    -   a Bluetooth indicator;    -   a Battery status indicator;    -   Tray 408 with icons for frequently used applications, such as:        -   Icon 416 for telephone module 138, labeled “Phone,” which            optionally includes an indicator 414 of the number of missed            calls or voicemail messages;        -   Icon 418 for e-mail client module 140, labeled “Mail,” which            optionally includes an indicator 410 of the number of unread            e-mails;        -   Icon 420 for browser module 147, labeled “Browser”; and        -   Icon 422 for video and music player module 152, labeled            “Music”; and    -   Icons for other applications, such as:        -   Icon 424 for IM module 141, labeled “Messages”;        -   Icon 426 for calendar module 148, labeled “Calendar”;        -   Icon 428 for image management module 144, labeled “Photos”;        -   Icon 430 for camera module 143, labeled “Camera”;        -   Icon 432 for measurement module 155, labeled “Measurement”;        -   Icon 434 for stocks widget 149-2, labeled “Stocks”;        -   Icon 436 for map module 154, labeled “Maps”;        -   Icon 438 for weather widget 149-1, labeled “Weather”;        -   Icon 440 for alarm clock widget 149-4, labeled “Clock”;        -   Icon 442 for workout support module 142, labeled “Workout            Support”;        -   Icon 444 for notes module 153, labeled “Notes”; and        -   Icon 446 for a settings application or module, labeled            “Settings,” which provides access to settings for device 100            and its various applications 136.

It should be noted that the icon labels illustrated in FIG. 4A aremerely examples. For example, other labels are, optionally, used forvarious application icons. In some embodiments, a label for a respectiveapplication icon includes a name of an application corresponding to therespective application icon. In some embodiments, a label for aparticular application icon is distinct from a name of an applicationcorresponding to the particular application icon.

FIG. 4B illustrates an example user interface on a device (e.g., device300, FIG. 3A) with a touch-sensitive surface 451 (e.g., a tablet ortouchpad 355, FIG. 3A) that is separate from the display 450. Althoughmany of the examples that follow will be given with reference to inputson touch screen display 112 (where the touch sensitive surface and thedisplay are combined), in some embodiments, the device detects inputs ona touch-sensitive surface that is separate from the display, as shown inFIG. 4B. In some embodiments, the touch-sensitive surface (e.g., 451 inFIG. 4B) has a primary axis (e.g., 452 in FIG. 4B) that corresponds to aprimary axis (e.g., 453 in FIG. 4B) on the display (e.g., 450). Inaccordance with these embodiments, the device detects contacts (e.g.,460 and 462 in FIG. 4B) with the touch-sensitive surface 451 atlocations that correspond to respective locations on the display (e.g.,in FIG. 4B, 460 corresponds to 468 and 462 corresponds to 470). In thisway, user inputs (e.g., contacts 460 and 462, and movements thereof)detected by the device on the touch-sensitive surface (e.g., 451 in FIG.4B) are used by the device to manipulate the user interface on thedisplay (e.g., 450 in FIG. 4B) of the multifunction device when thetouch-sensitive surface is separate from the display. It should beunderstood that similar methods are, optionally, used for other userinterfaces described herein.

Additionally, while the following examples are given primarily withreference to finger inputs (e.g., finger contacts, finger tap gestures,finger swipe gestures, etc.), it should be understood that, in someembodiments, one or more of the finger inputs are replaced with inputfrom another input device (e.g., a mouse based input or a stylus input).For example, a swipe gesture is, optionally, replaced with a mouse click(e.g., instead of a contact) followed by movement of the cursor alongthe path of the swipe (e.g., instead of movement of the contact). Asanother example, a tap gesture is, optionally, replaced with a mouseclick while the cursor is located over the location of the tap gesture(e.g., instead of detection of the contact followed by ceasing to detectthe contact). Similarly, when multiple user inputs are simultaneouslydetected, it should be understood that multiple computer mice are,optionally, used simultaneously, or a mouse and finger contacts are,optionally, used simultaneously.

As used herein, the term “focus selector” refers to an input elementthat indicates a current part of a user interface with which a user isinteracting. In some implementations that include a cursor or otherlocation marker, the cursor acts as a “focus selector,” so that when aninput (e.g., a press input) is detected on a touch-sensitive surface(e.g., touchpad 355 in FIG. 3A or touch-sensitive surface 451 in FIG.4B) while the cursor is over a particular user interface element (e.g.,a button, window, slider or other user interface element), theparticular user interface element is adjusted in accordance with thedetected input. In some implementations that include a touch-screendisplay (e.g., touch-sensitive display system 112 in FIG. 1A or thetouch screen in FIG. 4A) that enables direct interaction with userinterface elements on the touch-screen display, a detected contact onthe touch-screen acts as a “focus selector,” so that when an input(e.g., a press input by the contact) is detected on the touch-screendisplay at a location of a particular user interface element (e.g., abutton, window, slider or other user interface element), the particularuser interface element is adjusted in accordance with the detectedinput. In some implementations, focus is moved from one region of a userinterface to another region of the user interface without correspondingmovement of a cursor or movement of a contact on a touch-screen display(e.g., by using a tab key or arrow keys to move focus from one button toanother button); in these implementations, the focus selector moves inaccordance with movement of focus between different regions of the userinterface. Without regard to the specific form taken by the focusselector, the focus selector is generally the user interface element (orcontact on a touch-screen display) that is controlled by the user so asto communicate the user's intended interaction with the user interface(e.g., by indicating, to the device, the element of the user interfacewith which the user is intending to interact). For example, the locationof a focus selector (e.g., a cursor, a contact, or a selection box) overa respective button while a press input is detected on thetouch-sensitive surface (e.g., a touchpad or touch screen) will indicatethat the user is intending to activate the respective button (as opposedto other user interface elements shown on a display of the device). Insome embodiments, a focus indicator (e.g., a cursor or selectionindicator) is displayed via the display device to indicate a currentportion of the user interface that will be affected by inputs receivedfrom the one or more input devices.

As used in the specification and claims, the term “intensity” of acontact on a touch-sensitive surface refers to the force or pressure(force per unit area) of a contact (e.g., a finger contact or a styluscontact) on the touch-sensitive surface, or to a substitute (proxy) forthe force or pressure of a contact on the touch-sensitive surface. Theintensity of a contact has a range of values that includes at least fourdistinct values and more typically includes hundreds of distinct values(e.g., at least 256). Intensity of a contact is, optionally, determined(or measured) using various approaches and various sensors orcombinations of sensors. For example, one or more force sensorsunderneath or adjacent to the touch-sensitive surface are, optionally,used to measure force at various points on the touch-sensitive surface.In some implementations, force measurements from multiple force sensorsare combined (e.g., a weighted average or a sum) to determine anestimated force of a contact. Similarly, a pressure-sensitive tip of astylus is, optionally, used to determine a pressure of the stylus on thetouch-sensitive surface. Alternatively, the size of the contact areadetected on the touch-sensitive surface and/or changes thereto, thecapacitance of the touch-sensitive surface proximate to the contactand/or changes thereto, and/or the resistance of the touch-sensitivesurface proximate to the contact and/or changes thereto are, optionally,used as a substitute for the force or pressure of the contact on thetouch-sensitive surface. In some implementations, the substitutemeasurements for contact force or pressure are used directly todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is described in units corresponding to thesubstitute measurements). In some implementations, the substitutemeasurements for contact force or pressure are converted to an estimatedforce or pressure and the estimated force or pressure is used todetermine whether an intensity threshold has been exceeded (e.g., theintensity threshold is a pressure threshold measured in units ofpressure). Using the intensity of a contact as an attribute of a userinput allows for user access to additional device functionality that mayotherwise not be readily accessible by the user on a reduced-size devicewith limited real estate for displaying affordances (e.g., on atouch-sensitive display) and/or receiving user input (e.g., via atouch-sensitive display, a touch-sensitive surface, or aphysical/mechanical control such as a knob or a button).

In some embodiments, contact/motion module 130 uses a set of one or moreintensity thresholds to determine whether an operation has beenperformed by a user (e.g., to determine whether a user has “clicked” onan icon). In some embodiments, at least a subset of the intensitythresholds are determined in accordance with software parameters (e.g.,the intensity thresholds are not determined by the activation thresholdsof particular physical actuators and can be adjusted without changingthe physical hardware of device 100). For example, a mouse “click”threshold of a trackpad or touch-screen display can be set to any of alarge range of predefined thresholds values without changing thetrackpad or touch-screen display hardware. Additionally, in someimplementations a user of the device is provided with software settingsfor adjusting one or more of the set of intensity thresholds (e.g., byadjusting individual intensity thresholds and/or by adjusting aplurality of intensity thresholds at once with a system-level click“intensity” parameter).

As used in the specification and claims, the term “characteristicintensity” of a contact refers to a characteristic of the contact basedon one or more intensities of the contact. In some embodiments, thecharacteristic intensity is based on multiple intensity samples. Thecharacteristic intensity is, optionally, based on a predefined number ofintensity samples, or a set of intensity samples collected during apredetermined time period (e.g., 0.05, 0.1, 0.2, 0.5, 1, 2, 5, 10seconds) relative to a predefined event (e.g., after detecting thecontact, prior to detecting liftoff of the contact, before or afterdetecting a start of movement of the contact, prior to detecting an endof the contact, before or after detecting an increase in intensity ofthe contact, and/or before or after detecting a decrease in intensity ofthe contact). A characteristic intensity of a contact is, optionallybased on one or more of: a maximum value of the intensities of thecontact, a mean value of the intensities of the contact, an averagevalue of the intensities of the contact, a top 10 percentile value ofthe intensities of the contact, a value at the half maximum of theintensities of the contact, a value at the 90 percent maximum of theintensities of the contact, a value produced by low-pass filtering theintensity of the contact over a predefined period or starting at apredefined time, or the like. In some embodiments, the duration of thecontact is used in determining the characteristic intensity (e.g., whenthe characteristic intensity is an average of the intensity of thecontact over time). In some embodiments, the characteristic intensity iscompared to a set of one or more intensity thresholds to determinewhether an operation has been performed by a user. For example, the setof one or more intensity thresholds may include a first intensitythreshold and a second intensity threshold. In this example, a contactwith a characteristic intensity that does not exceed the first intensitythreshold results in a first operation, a contact with a characteristicintensity that exceeds the first intensity threshold and does not exceedthe second intensity threshold results in a second operation, and acontact with a characteristic intensity that exceeds the secondintensity threshold results in a third operation. In some embodiments, acomparison between the characteristic intensity and one or moreintensity thresholds is used to determine whether or not to perform oneor more operations (e.g., whether to perform a respective option orforgo performing the respective operation) rather than being used todetermine whether to perform a first operation or a second operation.

In some embodiments, a portion of a gesture is identified for purposesof determining a characteristic intensity. For example, atouch-sensitive surface may receive a continuous swipe contacttransitioning from a start location and reaching an end location (e.g.,a drag gesture), at which point the intensity of the contact increases.In this example, the characteristic intensity of the contact at the endlocation may be based on only a portion of the continuous swipe contact,and not the entire swipe contact (e.g., only the portion of the swipecontact at the end location). In some embodiments, a smoothing algorithmmay be applied to the intensities of the swipe contact prior todetermining the characteristic intensity of the contact. For example,the smoothing algorithm optionally includes one or more of: anunweighted sliding-average smoothing algorithm, a triangular smoothingalgorithm, a median filter smoothing algorithm, and/or an exponentialsmoothing algorithm. In some circumstances, these smoothing algorithmseliminate narrow spikes or dips in the intensities of the swipe contactfor purposes of determining a characteristic intensity.

The user interface figures described herein optionally include variousintensity diagrams that show the current intensity of the contact on thetouch-sensitive surface relative to one or more intensity thresholds(e.g., a contact detection intensity threshold IT₀, a light pressintensity threshold IT_(L), a deep press intensity threshold IT_(D)(e.g., that is at least initially higher than IT_(L)), and/or one ormore other intensity thresholds (e.g., an intensity threshold IT_(H)that is lower than IT_(L))). This intensity diagram is typically notpart of the displayed user interface, but is provided to aid in theinterpretation of the figures. In some embodiments, the light pressintensity threshold corresponds to an intensity at which the device willperform operations typically associated with clicking a button of aphysical mouse or a trackpad. In some embodiments, the deep pressintensity threshold corresponds to an intensity at which the device willperform operations that are different from operations typicallyassociated with clicking a button of a physical mouse or a trackpad. Insome embodiments, when a contact is detected with a characteristicintensity below the light press intensity threshold (e.g., and above anominal contact-detection intensity threshold IT₀ below which thecontact is no longer detected), the device will move a focus selector inaccordance with movement of the contact on the touch-sensitive surfacewithout performing an operation associated with the light pressintensity threshold or the deep press intensity threshold. Generally,unless otherwise stated, these intensity thresholds are consistentbetween different sets of user interface figures.

In some embodiments, the response of the device to inputs detected bythe device depends on criteria based on the contact intensity during theinput. For example, for some “light press” inputs, the intensity of acontact exceeding a first intensity threshold during the input triggersa first response. In some embodiments, the response of the device toinputs detected by the device depends on criteria that include both thecontact intensity during the input and time-based criteria. For example,for some “deep press” inputs, the intensity of a contact exceeding asecond intensity threshold during the input, greater than the firstintensity threshold for a light press, triggers a second response onlyif a delay time has elapsed between meeting the first intensitythreshold and meeting the second intensity threshold. This delay time istypically less than 200 ms (milliseconds) in duration (e.g., 40, 100, or120 ms, depending on the magnitude of the second intensity threshold,with the delay time increasing as the second intensity thresholdincreases). This delay time helps to avoid accidental recognition ofdeep press inputs. As another example, for some “deep press” inputs,there is a reduced-sensitivity time period that occurs after the time atwhich the first intensity threshold is met. During thereduced-sensitivity time period, the second intensity threshold isincreased. This temporary increase in the second intensity thresholdalso helps to avoid accidental deep press inputs. For other deep pressinputs, the response to detection of a deep press input does not dependon time-based criteria.

In some embodiments, one or more of the input intensity thresholdsand/or the corresponding outputs vary based on one or more factors, suchas user settings, contact motion, input timing, application running,rate at which the intensity is applied, number of concurrent inputs,user history, environmental factors (e.g., ambient noise), focusselector position, and the like. Example factors are described in U.S.patent application Ser. Nos. 14/399,606 and 14/624,296, which areincorporated by reference herein in their entireties.

For example, FIG. 4C illustrates a dynamic intensity threshold 480 thatchanges over time based in part on the intensity of touch input 476 overtime. Dynamic intensity threshold 480 is a sum of two components, firstcomponent 474 that decays over time after a predefined delay time p1from when touch input 476 is initially detected, and second component478 that trails the intensity of touch input 476 over time. The initialhigh intensity threshold of first component 474 reduces accidentaltriggering of a “deep press” response, while still allowing an immediate“deep press” response if touch input 476 provides sufficient intensity.Second component 478 reduces unintentional triggering of a “deep press”response by gradual intensity fluctuations of in a touch input. In someembodiments, when touch input 476 satisfies dynamic intensity threshold480 (e.g., at point 481 in FIG. 4C), the “deep press” response istriggered.

FIG. 4D illustrates another dynamic intensity threshold 486 (e.g.,intensity threshold ITS). FIG. 4D also illustrates two other intensitythresholds: a first intensity threshold IT_(H) and a second intensitythreshold IT_(L). In FIG. 4D, although touch input 484 satisfies thefirst intensity threshold IT_(H) and the second intensity thresholdIT_(L) prior to time p2, no response is provided until delay time p2 haselapsed at time 482. Also in FIG. 4D, dynamic intensity threshold 486decays over time, with the decay starting at time 488 after a predefineddelay time p1 has elapsed from time 482 (when the response associatedwith the second intensity threshold IT_(L) was triggered). This type ofdynamic intensity threshold reduces accidental triggering of a responseassociated with the dynamic intensity threshold IT_(D) immediatelyafter, or concurrently with, triggering a response associated with alower intensity threshold, such as the first intensity threshold IT_(H)or the second intensity threshold IT_(L).

FIG. 4E illustrate yet another dynamic intensity threshold 492 (e.g.,intensity threshold IT_(D)). In FIG. 4E, a response associated with theintensity threshold IT_(L) is triggered after the delay time p2 haselapsed from when touch input 490 is initially detected. Concurrently,dynamic intensity threshold 492 decays after the predefined delay timep1 has elapsed from when touch input 490 is initially detected. So adecrease in intensity of touch input 490 after triggering the responseassociated with the intensity threshold IT_(L), followed by an increasein the intensity of touch input 490, without releasing touch input 490,can trigger a response associated with the intensity threshold IT_(D)(e.g., at time 494) even when the intensity of touch input 490 is belowanother intensity threshold, for example, the intensity thresholdIT_(L).

An increase of characteristic intensity of the contact from an intensitybelow the light press intensity threshold IT_(L) to an intensity betweenthe light press intensity threshold IT_(L) and the deep press intensitythreshold IT_(D) is sometimes referred to as a “light press” input. Anincrease of characteristic intensity of the contact from an intensitybelow the deep press intensity threshold IT_(D) to an intensity abovethe deep press intensity threshold IT_(D) is sometimes referred to as a“deep press” input. An increase of characteristic intensity of thecontact from an intensity below the contact-detection intensitythreshold IT₀ to an intensity between the contact-detection intensitythreshold IT₀ and the light press intensity threshold IT_(L) issometimes referred to as detecting the contact on the touch-surface. Adecrease of characteristic intensity of the contact from an intensityabove the contact-detection intensity threshold IT₀ to an intensitybelow the contact-detection intensity threshold IT₀ is sometimesreferred to as detecting liftoff of the contact from the touch-surface.In some embodiments IT₀ is zero. In some embodiments, IT₀ is greaterthan zero. In some illustrations a shaded circle or oval is used torepresent intensity of a contact on the touch-sensitive surface. In someillustrations, a circle or oval without shading is used represent arespective contact on the touch-sensitive surface without specifying theintensity of the respective contact.

In some embodiments, described herein, one or more operations areperformed in response to detecting a gesture that includes a respectivepress input or in response to detecting the respective press inputperformed with a respective contact (or a plurality of contacts), wherethe respective press input is detected based at least in part ondetecting an increase in intensity of the contact (or plurality ofcontacts) above a press-input intensity threshold. In some embodiments,the respective operation is performed in response to detecting theincrease in intensity of the respective contact above the press-inputintensity threshold (e.g., the respective operation is performed on a“down stroke” of the respective press input). In some embodiments, thepress input includes an increase in intensity of the respective contactabove the press-input intensity threshold and a subsequent decrease inintensity of the contact below the press-input intensity threshold, andthe respective operation is performed in response to detecting thesubsequent decrease in intensity of the respective contact below thepress-input threshold (e.g., the respective operation is performed on an“up stroke” of the respective press input).

In some embodiments, the device employs intensity hysteresis to avoidaccidental inputs sometimes termed “jitter,” where the device defines orselects a hysteresis intensity threshold with a predefined relationshipto the press-input intensity threshold (e.g., the hysteresis intensitythreshold is X intensity units lower than the press-input intensitythreshold or the hysteresis intensity threshold is 75%, 90%, or somereasonable proportion of the press-input intensity threshold). Thus, insome embodiments, the press input includes an increase in intensity ofthe respective contact above the press-input intensity threshold and asubsequent decrease in intensity of the contact below the hysteresisintensity threshold that corresponds to the press-input intensitythreshold, and the respective operation is performed in response todetecting the subsequent decrease in intensity of the respective contactbelow the hysteresis intensity threshold (e.g., the respective operationis performed on an “up stroke” of the respective press input).Similarly, in some embodiments, the press input is detected only whenthe device detects an increase in intensity of the contact from anintensity at or below the hysteresis intensity threshold to an intensityat or above the press-input intensity threshold and, optionally, asubsequent decrease in intensity of the contact to an intensity at orbelow the hysteresis intensity, and the respective operation isperformed in response to detecting the press input (e.g., the increasein intensity of the contact or the decrease in intensity of the contact,depending on the circumstances).

For ease of explanation, the description of operations performed inresponse to a press input associated with a press-input intensitythreshold or in response to a gesture including the press input are,optionally, triggered in response to detecting: an increase in intensityof a contact above the press-input intensity threshold, an increase inintensity of a contact from an intensity below the hysteresis intensitythreshold to an intensity above the press-input intensity threshold, adecrease in intensity of the contact below the press-input intensitythreshold, or a decrease in intensity of the contact below thehysteresis intensity threshold corresponding to the press-inputintensity threshold. Additionally, in examples where an operation isdescribed as being performed in response to detecting a decrease inintensity of a contact below the press-input intensity threshold, theoperation is, optionally, performed in response to detecting a decreasein intensity of the contact below a hysteresis intensity thresholdcorresponding to, and lower than, the press-input intensity threshold.As described above, in some embodiments, the triggering of theseresponses also depends on time-based criteria being met (e.g., a delaytime has elapsed between a first intensity threshold being met and asecond intensity threshold being met).

Although only specific frequencies, amplitudes, and waveforms arerepresented in the sample tactile output patterns in FIGS. 4F-4K forillustrative purposes, tactile output patterns with other frequencies,amplitudes, and waveforms may be used for similar purposes. For example,waveforms that have between 0.5 to 4 cycles can be used. Otherfrequencies in the range of 60 Hz-400 Hz may be used as well.

User Interfaces and Associated Processes

Attention is now directed towards embodiments of user interfaces (“UI”)and associated processes that may be implemented on a computer system(e.g., an electronic device such as portable multifunction device 100(FIG. 1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B)) thatincludes (and/or is in communication with) a display generationcomponent (e.g., a display, a projector, a heads-up display, or thelike), one or more cameras (e.g., video cameras that continuouslyprovide a live preview of at least a portion of the contents that arewithin the field of view of at least one of the cameras and optionallygenerate video outputs including one or more streams of image framescapturing the contents within the field of view of at least one of thecameras), and one or more input devices (e.g., a touch-sensitivesurface, such as a touch-sensitive remote control, or a touch-screendisplay that also serves as the display generation component, a mouse, ajoystick, a wand controller, and/or cameras tracking the position of oneor more features of the user such as the user's hands), optionally oneor more attitude sensors, optionally one or more sensors to detectintensities of contacts with the touch-sensitive surface, and optionallyone or more tactile output generators.

FIGS. 5A-5CO illustrate example user interfaces for making measurementsof a physical space using an augmented reality environment in accordancewith some embodiments. The user interfaces in these figures are used toillustrate the processes described below, including the processes inFIGS. 6A-6C, 7A-7E, 8A-8C, 9A-9B, 10A-10B, 11A-11B, 12A-12C, 13A-13C,and 14A-14D. For convenience of explanation, some of the embodimentswill be discussed with reference to operations performed on a devicewith a touch-sensitive display system 112. In such embodiments, thefocus selector is, optionally: a respective finger or stylus contact, arepresentative point corresponding to a finger or stylus contact (e.g.,a centroid of a respective contact or a point associated with arespective contact), or a centroid of two or more contacts detected onthe touch-sensitive display system 112. However, analogous operationsare, optionally, performed on a device with a display 450 and a separatetouch-sensitive surface 451 in response to detecting the contacts on thetouch-sensitive surface 451 while displaying the user interfaces shownin the figures on the display 450, along with a focus selector.

FIG. 5A illustrates a context in which user interfaces described withrespect to FIGS. 5A-5CO are used. In particular, FIG. 5A illustrates aview of physical space 5000 in which table 5002 and device 100 arelocated. Device 100 is held by user 5004 to view physical space 5000,including a portion of table 5002, through touch screen 112 of device100. In particular, touch screen 112 displays a view of an augmentedreality environment corresponding to physical space 5000. User 5004 usestouch screen 112 of device 100 to interact with the augmented realityenvironment via displayed user interface 5006 (e.g., a user interface ofan augmented reality measurement application). User interface 5006includes a live preview of the field of view of at least one of one ormore cameras of device 100 (sometimes referred to as “the camera” ofdevice 100, such as camera(s) 305, FIGS. 3A-3B or camera(s) 311, FIG.3C, optionally including optical sensors 164, FIG. 1A as part of the oneor more cameras). In some embodiments, the camera(s) are located ondevice 100 in region 5008. In some embodiments, device 100 includesfront-facing camera 305-a that is located in region 5008 next to and onthe same side of device 100 as touch screen 112 (e.g., the side facinguser 5004 in FIG. 5A). In some embodiments, device 100 includes one ormore cameras that are located in region 5008 behind touch screen 112 oron the opposite side of device 100 (sometimes called the back side ofdevice 100) from touch screen 112 (e.g., the one or more cameras are onthe side facing away from user 5004 in FIG. 5A). At least one cameracontinuously provides a live preview of contents that are within thefield of view the camera, which may include one or more physical objectsin physical space 5000 (e.g., table 5002).

In some embodiments, user interface 5006 includes one or more userinterface elements for user interaction with the augmented realityenvironment. For example, in FIG. 5A, user interface 5006 includesreticle 5010 that indicates an area for user interaction with theaugmented reality environment. In some embodiments, reticle 5010includes focus point 5012 that indicates a particular point for userinteraction. In some embodiments, user interface 5006 includesmeasurement addition button 5014 that is used for adding newmeasurements (e.g., new measurement points, new measurement segments,and/or new measurement regions) to user interface 5006 (e.g., asdescribed in more detail herein). In some embodiments, reticle 5010 andfocus point 5012 together form a measurement-point-creation indicatorthat indicates a location at which a new measurement will be added inresponse to activation of measurement addition button 5014.

In some embodiments, user interface 5006 includes media capture button5016 that is used for capturing media, such as a still image, or a video(that optionally includes corresponding audio), of the field of view,and any virtual measurements corresponding to physical objects in thefield of view. In some embodiments, user interface 5006 includes undobutton 5018 that is used for undoing (e.g., reversing the performanceof) a most-recently-performed operation in user interface 5006. In someembodiments, user interface 5006 includes redo button 5020 that is usedfor redoing a most-recently-undone operation (e.g., reversing thereversal of the most-recently-performed operation by activation of undobutton 5018) in user interface 5006. User interface 5006 may alsoinclude one or more virtual measurements that correspond to one or morephysical objects in physical space 5000, and that are displayed at leastpartially in response to user inputs on the user interface elements ofuser interface 5006. In some embodiments, user interface 5006 includesclear button 5022 for removing virtual measurements displayed in userinterface 5006 (and, optionally, also removing virtual measurements thatare not displayed in user interface 5006 when clear button 5022 isactivated, such as virtual measurements corresponding to physicalobject(s) that are outside of the field of view of the camera when clearbutton 5022 is activated).

FIG. 5B illustrates device 100 in a first landscape orientation, incontrast to its portrait orientation in FIG. 5A. While device 100 is inthe first landscape orientation, as shown in FIG. 5B, the one or morecameras of device 100 are located in region 5008 on the left side ofdevice 100, and measurement addition button 5014 is displayed in userinterface 5006 on the right side of device 100, away from region 5008,to reduce the chance that user 5004 will hold device 100 on the leftside and obscure the field of view of the camera providing the livepreview in user interface 5006, and also to facilitate interaction withthe augmented reality environment during single-handed operation by user5004 while holding device 100 by its right side (e.g., while user 5004holds device 100 with his/her right hand).

FIG. 5C illustrates device 100 in a second landscape orientation,different from the landscape orientation in FIG. 5B (e.g., 180 degreesrotated from the landscape orientation in FIG. 5B). While device 100 isin the second landscape orientation, as shown in FIG. 5C, the one ormore cameras of device 100 are located in region 5008 on the right sideof device 100, and measurement addition button 5014 is displayed in userinterface 100 on the left side of device 100, away from region 5008, toreduce the chance that user 5004 will hold device 100 on the right sideand obscure the field of view of the camera providing the live previewin user interface 5006, and also to facilitate interaction with theaugmented reality environment during single-handed operation by user5004 while holding device 100 by its left side (e.g., while user 5004holds device 100 with his/her left hand).

FIG. 5D illustrates an example of an error condition of the augmentedreality environment. In particular, FIG. 5D shows a view of physicalspace 5000 when there is insufficient light available for device 100 torecognize distinct physical objects and physical features in the fieldof view of the camera. Device 100 displays error message 5024 (e.g.,with the text “Too dark” and “Turn on lights to detect surface”) in userinterface 5006 to indicate the error condition and to prompt user 5004to increase the amount of light in physical space 5000. In someembodiments, while an error condition exists such that device 100 isunable to identify a physical object or feature in the field of view ofthe camera, device 100 ceases to display reticle 5010 and/or focus point5012 to indicate the error condition.

FIG. 5E illustrates another example of an error condition of theaugmented reality environment. In particular, FIG. 5E shows a view ofphysical space 5000 when there is sufficient light available (e.g., user5004 has turned on the lights in response to error message 5024), butdevice 100 does not detect a surface of a physical object in physicalspace 5000 that is in the field of view of the camera. Device 100displays error message 5026 (e.g., with the text “Move device from sideto side to detect surface”) in user interface 5006 to indicate the errorcondition and to prompt user 5004 to move device 100 from side to side(e.g., to facilitate image processing by device 100 to detect a physicalsurface that is in the field of view of the camera). Movement arrows5028 indicate side-to-side movement of device 100 by user 5004 inresponse to error message 5026. In some embodiments, error message 5026is displayed when device 100 does not detect a surface of a physicalobject at the location over which focus point 5012 is displayed in thelive preview. In some embodiments, a different error message isdisplayed if user 5004 moves device 100 too quickly for a surface of aphysical object to be detected, to indicate the error condition and toprompt user 5004 to move device 100 more slowly.

FIG. 5F illustrates a view of physical space 5000 when device 100 hasdetected a surface of physical object in physical space 5000. Inparticular, in FIG. 5F, device 100 has detected the top surface of table5002 (e.g., based on the side-to-side movement of device 100, asdescribed above with respect to FIG. 5E, while focus point 5012 ispositioned over the top surface of table 5002 in the live preview). Inaccordance with detecting a surface, device 100 displays reticle 5010 inuser interface 5006, in addition to focus point 5012. In someembodiments, reticle 5010 is displayed whenever focus point 5012 ispositioned over a region in the live preview that corresponds to adetected surface of a physical object in physical space 5000 (e.g., toindicate that the region in the live preview over which focus point 5012is positioned corresponds to a detected surface). In some embodiments,reticle 5010 is tilted to appear to be co-planar with the detectedsurface, as illustrated in FIG. 5F, to indicate the surface that hasbeen detected by the device.

FIG. 5G illustrates a transition from FIG. 5F. In FIG. 5G, user 5004 hasmoved device 100 such that focus point 5012 is positioned over a pointin the live preview that does not correspond to a detected surface(e.g., focus point 5012 is no longer positioned over the detected topsurface of table 5002 in the live preview). Accordingly, device 100changes the appearance of reticle 5010 (e.g., by no longer tilting thereticle 5010, by changing the reticle to a solid circle, and/or byceasing to display the reticle), while continuing to display focus point5012.

FIG. 5H illustrates a transition from FIG. 5G. In FIG. 5H, user 5004 hasmoved device 100 such that focus point 5012 and at least part of reticle5010 are repositioned over the detected top surface of table 5002 in thelive preview. Device 100 has detected edge 5030 of table 5002 in thelive preview (e.g., an edge of the detected top surface). Edge 5030 isat least partially within reticle 5010 (e.g., focus point 5012 is withina predefined threshold distance of edge 5030, where the predefinedthreshold distance is at most the radius of reticle 5010). Accordingly,the visual appearances of reticle 5010 and focus point 5012 are changed.In particular, focus point 5012 has been moved (e.g., verticallydownward) to, or “snapped” to, a point along edge 5030. In addition, asize of reticle 5010 is reduced to indicate that focus point 5012 hassnapped to a detected feature in the live preview. In some embodiments,reticle 5010 is displayed at the size shown in FIG. 5H whenever focuspoint 5012 is snapped to a detected feature in the live preview. In someembodiments, the visual appearances of reticle 5010 and focus point 5012are changed when focus point 5012 snaps to a detected feature in thelive preview that corresponds to an edge or a corner of a physicalobject in the field of view of the camera. In some embodiments, thevisual appearances of reticle 5010 and focus point 5012 are not changedwhen reticle 5010 and focus point 5012 are positioned over a detectedfeature in the live preview that corresponds to a surface of a physicalobject in the field of view of the camera (but not an edge or a corner).In some embodiments, in conjunction with moving focus point 5012 to edge5030, device 100 optionally generates tactile output 5032 (e.g., usingtactile output generators 163, FIG. 1A) to indicate that point 5012 hasbeen “snapped” to a detected feature in the live preview of the camera.

FIG. 5I illustrates a transition from FIG. 5H. In FIG. 5I, user 5004 hasmoved device 100 such that no portion of edge 5030 is within reticle5010. Accordingly, focus point 5012 is snapped away from edge 5030 andis redisplayed at the center of reticle 5010 (e.g., because there is nolonger any feature within reticle 5010 to which focus point 5012 shouldsnap). In addition, the size of reticle 5010 is increased back to itssize as shown in FIG. 5F, prior to focus point 5012 snapping to edge5030. In some embodiments, reticle 5010 is displayed at the size shownin FIGS. 5F and 51 by default, whenever focus point 5012 is not snappedto any detected feature in the live preview. In some embodiments, focuspoint 5012 is displayed at the center of reticle 5010 by default,whenever focus point 5012 is not snapped to any detected feature in thelive preview. In some embodiments, as shown in FIG. 5I, device 100optionally generates tactile output 5034 in conjunction with movingfocus point 5012 away from edge 5030, to indicate focus point 5012 is nolonger snapped to a detected feature in the live preview of the camera.In some embodiments, tactile output 5034 differs from tactile output5032 (FIG. 5H) in at least one tactile output property (e.g., frequency,amplitude, and/or pattern), such that tactile output 5032 (FIG. 5H),which indicates snapping to a detected feature, provides a differenttactile feedback to user 5004 than tactile output 5034, which indicatessnapping away from a detected feature.

FIGS. 5J-5O illustrate creation of a measurement of the horizontalupper-left edge of table 5002. FIG. 5J illustrates a transition fromFIG. 5I. In FIG. 5J, user 5004 has moved device 100 such that theupper-back-left corner of table 5002, as displayed in the live preview,is within reticle 5010. Accordingly, focus point 5012 is snapped to theanchor point corresponding to the upper-back-left corner of table 5002in the live preview. In some embodiments, focus point 5012 is maintainedon the anchor point in the live preview as long as the upper-back-leftcorner of table 5002 in the live preview is within reticle 5010 (e.g.,although device 100 may move slightly due to unintentional movements ofuser 5004, such as unsteadiness of user 5004's hands). The size ofreticle 5010 is decreased to indicate the snapping behavior (e.g., tothe same size shown in and described above with reference to FIG. 5H).In addition, tactile output 5036 is generated to indicate the snappingbehavior.

FIGS. 5K-5L illustrate a transition from FIG. 5J showing the addition ofa measurement point to user interface 5006. In particular, FIG. 5Killustrates activation of measurement addition button 5014 by touchinput 5038 (e.g., a tap gesture) with a contact intensity that is abovea minimum contact detection threshold IT₀, as indicated by intensitymeter 5040. In response to the activation of measurement addition button5014, device 100 adds and displays (virtual) measurement point 5042 touser interface 5006 at a current location of focus point 5012. Inconjunction with adding measurement point 5042 to user interface 5006,device 100 optionally generates tactile output 5044 to indicate theaddition of a measurement point. In some embodiments, tactile output5044 differs from tactile output 5032 (FIG. 5H) and tactile output 5034(FIG. 5I) in at least one tactile output property (e.g., frequency,amplitude, and/or pattern), such that tactile output 5032 (FIG. 5H) andtactile output 5034 (FIG. 5I), which indicate snapping behavior, providedifferent tactile feedback to user 5004 than tactile output 5044, whichindicates the addition of a measurement point. FIG. 5L illustratesliftoff of touch input 5038 from measurement addition button 5014.

FIG. 5M illustrates a transition from FIG. 5L. In FIG. 5M, user 5004 hasmoved device 100 diagonally downward and toward the right such thatreticle 5010 is positioned over a different location in physical space5000 as displayed in the live preview. In particular, in FIG. 5M,reticle 5010 is positioned over the upper-front-left corner of table5002 as displayed in the live preview. Accordingly, focus point 5012 issnapped to the anchor point corresponding to the upper-front-left cornerof table 5002 in the live preview. The size of reticle 5010 is decreasedto indicate the snapping behavior (e.g., to the same size shown in anddescribed above with reference to FIG. 5J). Tactile output 5046 isgenerated to indicate the snapping behavior. In addition, measurementpoint 5042 continues to be displayed over the upper-back-left corner oftable 5002 in the live preview (e.g., measurement point 5042 isassociated with the upper-back-left corner of table 5002 and isdisplayed over that position in the live preview even while the positionof the upper-back-left corner of table 5002 in the live preview changesas device 100 moves).

In response to the movement of device 100 such that reticle 5010 andfocus point 5012 are positioned over a different location in physicalspace 5000, measurement segment 5048 is displayed between measurementpoint 5042 (the most-recently-added measurement point) and focus point5012. Measurement segment 5048 is displayed with label 5049 thatindicates a distance between the point in physical space 5000corresponding to measurement point 5042 and the point in physical space5000 corresponding to focus point 5012 (e.g., “3 ft”). Before the secondendpoint of measurement segment 5048 is added, measurement segment 5048is a dynamic measurement segment having a first endpoint that ismeasurement point 5042 and a second endpoint that is the currentposition of focus point 5012 (e.g., the length of measurement segment5048 and the distance indicated by label 5049 corresponding tomeasurement segment 5048 are both updated in accordance with movement ofdevice 100 that changes the point in physical space 5000 to which thecurrent location of focus point 5012 corresponds). In addition, virtualguide 5050 is displayed along a feature in physical space 5000 thatextends in the direction of movement of device 100 from themost-recently-added measurement point, measurement point 5042.Specifically, virtual guide 5050 is displayed along the horizontalupper-left edge of table 5002 that extends diagonally downward andtoward the right in the live preview. Note that the upper-left edge oftable 5002, and other edges of the top surface of table 5002, arereferred to herein as horizontal edges, because they are horizontal inphysical space 5000 even though they may appear diagonal in the livepreview, from the perspective of device 100.

FIGS. 5N-5O illustrate a transition from FIG. 5M showing the addition ofa measurement point to user interface 5006. In particular, FIG. 5Nillustrates activation of measurement addition button 5014 by touchinput 5052 (e.g., a tap gesture) with a contact intensity that is abovea minimum contact detection threshold IT₀, as indicated by intensitymeter 5040. In response to the activation of measurement addition button5014, device 100 adds and displays (virtual) measurement point 5054 inuser interface 5006 at a current location of focus point 5012 and as thesecond endpoint of measurement segment 5048. In conjunction with addingmeasurement point 5040 to user interface 5006, device 100 optionallygenerates tactile output 5056 to indicate the addition of a measurementpoint. In some embodiments, tactile output 5056 is the same type oftactile output as tactile output 5044 (FIG. 5K), indicating the additionof a measurement point. In some embodiments, tactile output 5056 differsfrom tactile output 5044 (FIG. 5K) in at least one tactile outputproperty (e.g., frequency, amplitude, and/or pattern), such that tactileoutput 5056 provides different tactile feedback to user 5004 thantactile output 5044 (FIG. 5K). In some such embodiments, tactile output5044 (FIG. 5K) indicates the addition of a measurement point that beginsa new measurement segment, while tactile output 5056 indicates theaddition of a measurement point that completes (e.g., ends) ameasurement segment. FIG. 5O illustrates liftoff of touch input 5052from measurement addition button 5014. After liftoff of touch input5052, device 100 continues to display measurement point 5042,measurement point 5054, measurement segment 5048 connecting measurementpoints 5042 and 5054 in user interface 5006, and label 5049(corresponding to a measurement of the horizontal upper-left edge oftable 5002). In some embodiments, as shown in FIG. 5O, upon thecompletion of a measurement segment, device 100 ceases to displayvirtual guides such as virtual guide 5050.

FIG. 5P illustrates a transition from FIG. 5O. In FIG. 5P, user 5004 hasmoved device 100 such that reticle 5010 is positioned over a sidesurface of table 5002 in the live preview that is adjacent tomeasurement segment 5048. Device 100 has determined (e.g., based in parton measurement segment 5048) that the region over which reticle 5010 (ormore specifically, focus point 5012) is positioned corresponds to aphysical rectangular area in physical space 5000 (e.g., although thedetected region appears trapezoidal in the live preview, from theperspective of device 100). Accordingly, device 100 displays indicator5058 to indicate that the detected region corresponds to a physicalrectangular area. In addition, device 100 continues to displaymeasurement point 5042, measurement point 5054, measurement segment5048, and label 5049 over the horizontal upper-left edge of table 5002.In some embodiments, measurement elements (e.g., measurement point 5042,measurement 5054, measurement segment 5048, and label 5049) that arepart of a virtual measurement are displayed in user interface 5006whenever the physical object corresponding to the virtual measurement(e.g., the horizontal upper-left edge of table 5002) is visible in thelive preview, until the corresponding virtual measurement is cleared.

FIG. 5Q-5X illustrate creation of a measurement of the verticalfront-left edge of table 5002. FIG. 5Q illustrates a transition fromFIG. 5P. In FIG. 5Q, user 5004 has moved device 100 such that reticle5010 is repositioned over the upper-front-left corner of table 5002 asdisplayed in the live preview. Accordingly, focus point 5012 is snappedto the anchor point corresponding to the upper-front-left corner oftable 5002 in the live preview, and corresponding to measurement point5054. The size of reticle 5010 is decreased to indicate the snappingbehavior (e.g., to the same size shown in and described above withreference to FIG. 5M). Tactile output 5060 is generated to indicate thesnapping behavior. Device 100 ceases to display indicator 5058 (e.g.,because focus point 5012 has snapped to the anchor point correspondingto the upper-front-left corner of table 5002, and is no longer displayedover the detected region in the live preview that corresponds to thephysical rectangular area of the side surface of table 5002).

FIG. 5R illustrates a transition from FIG. 5Q showing the addition of ameasurement point to user interface 5006. In particular, FIG. 5Rillustrates activation of measurement addition button 5014 by touchinput 5062 (e.g., a tap gesture) with a contact intensity that is abovea minimum contact detection threshold IT₀, as indicated by intensitymeter 5040. In some embodiments, as shown in FIG. 5R, in response to theactivation of measurement addition button 5014, device 100 usesmeasurement point 5054 as the starting point for a new measurementsegment to be added. In some embodiments, in response to the activationof measurement addition button 5014, device 100 adds and displaysanother (virtual) measurement point, at the same location as measurementpoint 5054, as the starting point for a new measurement segment. Inconjunction with establishing either of the aforementioned startingpoints for a new measurement segment, device 100 optionally generatestactile output 5064 to indicate the addition of a measurement point(and, in some embodiments, to indicate that the added measurement pointbegins a new measurement segment).

FIG. 5S illustrates a transition from FIG. 5R. In FIG. 5S, user 5004 hasmoved device 100 horizontally toward the right such that reticle 5010 ispositioned over a location along the horizontal front edge of table5002, and such that the upper-front-left corner of table 5002 in thelive preview is no longer within reticle 5010. Accordingly, focus point5012 is displayed on an anchor point corresponding to the horizontalfront edge of table 5002 (e.g., at the midpoint of the portion of thefront edge of table 5002 that is within reticle 5010, which is the pointalong the portion of the front edge of table 5002 within reticle 5010that is the shortest distance from the center of reticle 5010). Reticle5010 is displayed at its decreased size (e.g., at the same size as inFIG. 5R) to indicate that focus point 5012 is snapped to a detectedfeature in the live preview. In addition, measurement point 5054continues to be displayed over the upper-front-left corner of table 5002in the live preview, and dynamic measurement segment 5066 is displayedbetween measurement point 5054 and focus point 5012. Label 5068indicates a distance between the point in physical space 5000corresponding to measurement point 5054 and the point in physical space5000 corresponding to focus point 5012. In addition, in response to thehorizontal movement of device 100, device 100 displays virtual guide5070 extending horizontally from measurement point 5054.

In some embodiments, as shown in FIG. 5S, a tactile output is notgenerated to indicate the snapping behavior. For example, while at leastsome portion of the horizontal front edge of table 5002 is beenmaintained within reticle 5010, such that focus point 5012 has notsnapped away from the horizontal front edge of table 5002 at any timeduring the movement of device 100, although the particular anchor pointto which focus 5012 is snapped changes while device 100 moves, device100 forgoes generating a tactile output so as to avoid continuousgeneration of tactile outputs as focus point 5012 moves along thedetected edge.

FIG. 5T illustrates a transition from FIG. 5S. In FIG. 5T, the directionof movement of device 100 has changed from horizontally toward the rightto vertically downward. In response to the change in direction ofmovement from horizontal movement to vertical movement, device 100ceases to display (horizontal) virtual guide 5070 (FIG. 5S) and insteaddisplays (vertical) virtual guide 5072 extending vertically frommeasurement point 5054 in the direction of movement of device 100. Focuspoint 5012 is snapped to an anchor point corresponding to verticalinner-left edge 5074 of table 5002, and in some embodiments, as shown inFIG. 5T, device 100 generates tactile output 5076 to indicate thesnapping behavior. Dynamic measurement segment 5066 is continuallyupdated in accordance with the movement of device 100 so as to bedisplayed between measurement point 5054 and a current location of focuspoint 5012. In addition, label 5068 is continually updated in accordancewith the movement of device and the updating of dynamic measurementsegment 5066 so that label 5068 is displayed at a midpoint of dynamicmeasurement segment 5066.

FIG. 5U illustrates a transition from FIG. 5T showing the addition of ameasurement point to user interface 5006. In particular, FIG. 5Uillustrates activation of measurement addition button 5014 by touchinput 5078 (e.g., a tap gesture) with a contact intensity that is abovea minimum contact detection threshold IT₀, as indicated by intensitymeter 5040. In response to the activation of measurement addition button5014, device 100 adds and displays (virtual) measurement point 5080 touser interface 5006 at a current location of focus point 5012 and as thesecond endpoint of measurement segment 5066, which becomes a completedmeasurement segment (e.g., whose second endpoint is now measurementpoint 5080 and no longer focus point 5012) rather than a dynamicmeasurement segment. In conjunction with adding measurement point 5040to user interface 5006, device 100 optionally generates tactile output5082 indicating the addition of a measurement point (and optionally toindicate in particular the addition of a measurement point thatcompletes a measurement segment, for example using the same type oftactile output as tactile output 5056, FIG. 5N). In some embodiments, asshown in FIG. 5U, device 100 continues to display virtual guides such asvirtual guide 5072 even after the completion of a measurement segment(e.g., until the device 100 is moved away from its current position).

FIG. 5V illustrates a transition from FIG. 5U that includes performingan “undo” operation to reverse the most-recently-performed operation inuser interface 5006. In particular, FIG. 5V illustrates activation ofundo button 5018 by touch input 5084 to reverse the addition ofmeasurement point 5080 (FIG. 5U). Accordingly, measurement point 5080 isremoved from user interface 5006. Measurement segment 5066 is now(again) a dynamic measurement segment whose second endpoint is focuspoint 5012 and that is updated as focus point 5012 moves.

FIG. 5W illustrates a transition from FIG. 5V. In FIG. 5W, user 5004 hasmoved device 100 downward such that reticle 5010 is positioned over thelower-front-left corner of table 5002 as displayed in the live preview.Accordingly, focus point 5012 is snapped to the anchor pointcorresponding to the lower-front-left corner of table 5002 (andoptionally also to virtual guide 5072). Reticle 5010 is displayed at itsdecreased size (e.g., at the same size as in FIG. 5U) to indicate thatfocus point 5012 is snapped to a detected feature in the live preview.Tactile output 5086 is generated to indicate the snapping behavior. InFIG. 5X, device 100 has moved such that only a portion of measurementsegment 5048 is displayed in user interface 5006, because theupper-back-left corner of table 5002 is no longer visible in the livepreview and thus only a portion of the horizontal upper-left edge oftable 5002 is visible in the live preview. Accordingly, label 5049 isdisplayed at a midpoint of only the displayed portion of measurementsegment 5048, rather than at the midpoint of the entire measurementsegment 5048 (e.g., as shown in FIG. 5V).

FIG. 5X illustrates a transition from FIG. 5W showing the addition of ameasurement point to user interface 5006. In particular, FIG. 5Xillustrates activation of measurement addition button 5014 by touchinput 5088 (e.g., a tap gesture) with a contact intensity that is abovea minimum contact detection threshold IT₀, as indicated by intensitymeter 5040. In response to the activation of measurement addition button5014, device 100 adds and displays (virtual) measurement point 5090 touser interface 5006 at a current location of focus point 5012 and as thesecond endpoint of measurement segment 5066, which becomes a completedmeasurement segment. In conjunction with adding measurement point 5090to user interface 5006, device 100 optionally generates tactile output5092 to indicate the addition of a measurement point (and optionally toindicate the addition of a measurement point that completes ameasurement segment, for example using the same type of tactile outputas tactile output 5056, FIG. 5N, or tactile output 5082, FIG. 5U).

FIGS. 5Y-5AF illustrate the creation of measurement regionscorresponding to physical rectangular areas in physical space 5000(e.g., surfaces of table 5002). In particular, FIGS. 5Y-5Z illustratethe creation of a measurement region corresponding to a physicalrectangular area that is displayed in its entirety in the live previewat one time. FIG. 5Y illustrates a transition from FIG. 5X. In FIG. 5Y,user 5004 has moved device 100 from its position in FIG. 5X such thatreticle 5010 is positioned over the side surface of table 5002 in thelive preview that is adjacent to both measurement segment 5048 and tomeasurement segment 5066. Device 100 has determined that the region inthe live preview over which reticle 5010 (or more specifically, focuspoint 5012) is positioned corresponds to a physical rectangular area inphysical space 5000 (e.g., the determination is based in part onmeasurement segment 5048 and measurement segment 5066 being adjacentsides of the detected region). Accordingly, device 100 displaysindicator 5094 to indicate that the detected region in the live previewover which focus point 5012 is positioned corresponds to a physicalrectangular area. In addition, device 100 continues to displaymeasurement segment 5048, corresponding label 5049, measurement segment5066, corresponding label 5068, and endpoints 5042, 5054, and 5090. FIG.5Y is similar to FIG. 5P, except that FIG. 5Y includes (a second)measurement segment 5066, with endpoints 5054 and 5090 and label 5068,in addition to (a first) measurement segment 5048, with endpoints 5042and 5054 and label 5049.

FIG. 5Z illustrates a transition from FIG. 5Y showing the addition of ameasurement region to user interface 5006. In particular, FIG. 5Zillustrates activation of measurement addition button 5014, while focuspoint 5012 is over the detected region indicated by indicator 5094, bytouch input 5096 (e.g., a tap gesture) with a contact intensity that isabove a minimum contact detection threshold IT₀, as indicated byintensity meter 5040. In response to the activation of measurementaddition button 5014, device 100 adds a measurement corresponding to thedetected region and changes the appearance of indicator 5094 to indicatethat the detected region has been confirmed as a measurement region. Inaddition, device 100 displays label 5098 that indicates an area of thephysical rectangular area corresponding to the confirmed region (e.g.,“7.5 ft²”). Optionally, in some embodiments, label 5098 is displayedwith indicator 5094 prior to the detected region being confirmed as ameasurement region (e.g., in some embodiments, label 5098 is displayedwith indicator 5094 in FIG. 5Y).

FIG. 5AA illustrates a transition from FIG. 5Z. In FIG. 5AA, user 5004has moved device 100 vertically upwards such that reticle 5010 ispositioned over the horizontal upper-left edge of table 5002 in the livepreview and measurement segment 5048. In particular, the midpoint ofmeasurement segment 5048 is within reticle 5010. In some embodiments, asshown in FIG. 5AA, the midpoint of a respective measurement segmentdisplayed in user interface 5006 can be an anchor point to which focuspoint 5012 snaps. Accordingly, in FIG. 5AA, focus point 5012 is snappedto the anchor point corresponding to the midpoint of measurement segment5048 (and also corresponding to the midpoint of the horizontalupper-left edge of table 5002). The size of reticle 5010 is decreased toindicate the snapping behavior, as previously described. Tactile output5100 is generated to indicate the snapping behavior. In addition, device100 continues to display confirmed-region indicator 5094 over thedetected region that corresponds to the physical rectangular area inphysical space 5000 (e.g., confirmed-region indicator 5094 is associatedwith the detected region that corresponds to the physical rectangulararea and is displayed over the detected region in the live preview evenwhile the detected region moves in the live preview as device 100moves).

FIG. 5AB-5AE illustrate creation of a measurement region correspondingto a physical rectangular area in physical space 5000 that is partiallyvisible in the live preview but not visible in its entirety at one time(e.g., a physical rectangular area of which only three sides, in wholeor in part, are visible in the live preview). FIG. 5AB illustrates atransition from FIG. 5AA. In FIG. 5AB, user 5004 has moved device 100horizontally toward the right such that reticle 5010 is positioned overa region in the live preview that is adjacent to measurement segment5048 and that corresponds to a partial view of the top surface of table5002. In particular, the left edge of the top surface of table 5002 isvisible, a portion of the front edge of the top surface of table 5002 isvisible, and a portion of the back edge of the top surface of table 5002is visible, whereas the right edge of the top surface of table 5002 isnot visible. Focus point 5012 has snapped away from the anchor pointcorresponding to the midpoint of measurement 5048 and is redisplayed atthe center of reticle 5010. Reticle 5010 is redisplayed at its increasedsize, and tactile output 5102 is optionally generated in conjunctionwith focus point 5012 moving away from the anchor point. In addition,device 100 has determined (e.g., based in part on measurement segment5048) that the region over which reticle 5010 (or more specifically,focus point 5012) is positioned corresponds to a portion of a physicalrectangular area in physical space 5000 (e.g., although the detectedregion does not appear rectangular in the live preview, from theperspective of device 100). Accordingly, device 100 displays indicator5104 to indicate that the detected region corresponds to a physicalrectangular area. Optionally, in accordance with the movement of device100, virtual guides 5106 are displayed. Virtual guides 5106 extendhorizontally and correspond to features in the live preview identifiedas extending horizontally, in the direction of movement of device 100.In some embodiments, where multiple features extend in the direction ofmovement of device 100, multiple virtual guides that extend in thedirection of movement are displayed. In some embodiments, virtual guidesthat extend in the direction of movement are displayed only for featuresover which corresponding measurement points have already been added inuser interface 5006.

FIGS. 5AC-5AD illustrate further horizontal movement of device 100 whilereticle 510 is positioned over the top surface of table 5002 in the livepreview. In FIG. 5AC, a portion of the front edge and a portion of theback edge of the top surface of table 5002 are visible, whereas neitherthe left edge nor the right edge of the top surface of table 5002 arevisible. In FIG. 5AD, the right edge of the top surface of table 5002 isnow visible, in addition to a portion of the front edge and a portion ofthe back edge. Device 100 continues to display indicator 5104 over thevisible portion of the top surface of table 5002 in the live preview aswell as horizontal virtual guides in accordance with the horizontalmovement of device 100.

FIG. 5AE illustrates a transition from FIG. 5AD showing the addition ofa measurement region to user interface 5006. In particular, FIG. 5AEillustrates activation of measurement addition button 5014, while focuspoint 5012 is over the detected region indicated by indicator 5104, bytouch input 5108 (e.g., a tap gesture) with a contact intensity that isabove a minimum contact detection threshold IT₀, as indicated byintensity meter 5040. In response to the activation of measurementaddition button 5014, device 100 adds a measurement corresponding to theentire detected region, although only a portion of the detected regionis visible, and changes the appearance of indicator 5104 to indicatethat the detected region has been confirmed as a measurement region. Inaddition, device 100 displays label 5110, label 5112, and label 5114.Label 5110 indicates an area of the (entire) physical rectangular areacorresponding to the (entire) confirmed region. Label 5112 indicates alength of a first side of the physical rectangular area corresponding tothe confirmed region (e.g., the right edge of the top surface of table5002). Label 5114 indicates a length of a second side, adjacent to thefirst side, of the physical rectangular area corresponding to theconfirmed region (e.g., the front edge of the top surface of table5002). Optionally, in some embodiments, label 5114 is displayed withindicator 5104 prior to the detected region being confirmed as ameasurement region (e.g., in some embodiments, label 5114 is displayedwith indicator 5104 in FIG. 5AD).

FIG. 5AF illustrates a transition from FIG. 5AE. In FIG. 5AF, user 5004has moved device 100 horizontally toward the left so that the leftportion of table 5002 is visible in the live preview. Indicator 5094,measurement segment 5048, and measurement segment 5066 (along with theirassociated labels) are displayed in user interface 5006 at theirrespective locations corresponding to respective features of table 5002(e.g., the side surface, the horizontal upper-left edge, and thevertical front-left edge, respectively, of table 5002), even thoughdevice 100 previously moved away from a position at which these elementsand the corresponding features of table 5002 were visible in userinterface 5006 and has now been moved back to a position (such as thatshown in FIG. 5AF) at which these elements and the correspondingfeatures of table 5002 are (again) visible in user interface 5006. Inaddition, indicator 5104 (along with its associated labels) is displayedover the visible portion of the top surface of table 5002, even thoughthe detected region corresponding to the top surface of table 5002 wasconfirmed while device 100 displayed a different portion of the topsurface of table 5002 that is not currently visible in user interface5006.

FIG. 5AG illustrates clearing from user interface 5006 all measurementsdisplayed over the live preview of and corresponding to physical space5000 (e.g., including measurements corresponding to physical objects notcurrently in the field of view of the camera). In FIG. 5AG, in responseto activation of clear button 5022 by touch input 5116, measurementsegments 5048 and 5066, measurement points 5042, 5054, and 5090,indicators 5094 and 5104, and all corresponding labels are removed fromuser interface 506.

FIGS. 5AH-5AS illustrate continuous creation of measurements based onchanges in intensity of a continuous touch input. In particular, FIGS.5AH-5AM illustrate creation of a measurement of the horizontalupper-left edge of table 5002. FIG. 5AH optionally illustrates atransition from FIG. 5AG. In FIG. 5AH, user 5004 has moved device 100such that the upper-back-left corner of table 5002, as displayed in thelive preview, is within reticle 5010. Accordingly, focus point 5012 issnapped to the anchor point corresponding to the upper-back-left cornerof table 5002 in the live preview. Reticle 5010 is displayed at itsdecreased size to indicate the snapping behavior. In addition, tactileoutput 5118 is optionally generated to indicate the snapping behavior.

FIG. 5AI illustrates a transition from FIG. 5AH showing the addition ofa measurement point to user interface 5006. In particular, FIG. 5AIillustrates activation of measurement addition button 5014 by touchinput 5120 (e.g., a light press gesture) with a contact intensity thatis above light press intensity threshold IT_(L), as indicated byintensity meter 5040. In response to the activation of measurementaddition button 5014, device 100 adds and displays (virtual) measurementpoint 5122 to user interface 5006 at the current location of focus point5012. In conjunction with adding measurement point 5122 to userinterface 5006, and in response to the increase in contact intensity oftouch input 5120 above light press intensity threshold IT_(L), device100 optionally generates tactile output 5124 to indicate the addition ofa measurement point. In some embodiments, as shown in FIG. 5AI, the sizeof measurement addition button 5014 is decreased as the contactintensity of touch input 5120 increases (e.g., measurement additionbutton 5014 is smaller in FIG. 5AI, when the contact intensity of touchinput 5120 is above light press intensity threshold IT_(L), than in FIG.5K, when the contact intensity of touch input 5038 is between minimumcontact detection threshold IT₀ and light press intensity thresholdIT_(L)).

FIG. 5AJ illustrates a transition from FIG. 5AI showing that touch input5120 is maintained on measurement addition button 5014 with a contactintensity that is above minimum contact detection threshold IT₀ but thathas decreased below light press intensity threshold IT_(L). In someembodiments, device 100 generates a tactile output upon detecting thedecrease in the contact intensity of touch input 5120 below light pressintensity threshold IT_(L) (e.g., tactile output 5124 is not generatedin response to the increase in contact intensity of touch input 5120above light press intensity threshold IT_(L), as shown in FIG. 5AI, butinstead is generated in response to the decrease in contact intensity oftouch input 5120 below light press intensity threshold IT_(L), as shownin FIG. 5AJ).

FIG. 5AK illustrates a transition from FIG. 5AJ. In FIG. 5AK, user 5004has moved device 100 diagonally downward and toward the right whilemaintaining touch input 5120 on measurement addition button 5014 with acontact intensity that is above minimum contact detection threshold IT₀and below light press intensity threshold IT_(L). Device 100 has beenmoved such that reticle 5010 is positioned over the upper-front-leftcorner of table 5002 as displayed in the live preview. Accordingly,focus point 5012 is snapped to the anchor point corresponding to theupper-front-left corner of table 5002 in the live preview. Reticle 5010is displayed at its decreased size to indicate the snapping behavior,and, optionally, tactile output 5126 is generated to indicate thesnapping behavior. Measurement point 5122 continues to be displayed overthe upper-back-left corner of table 5002 in the live preview, anddynamic measurement segment 5128 is displayed between the currentposition of focus point 5012 and measurement point 5122. In addition, inaccordance with the movement of device 100, virtual guide 5130 isdisplayed, where virtual guide 5130 extends diagonally from measurementpoint 5122 and along the horizontal upper-left edge of table 5002 inuser interface 5006.

FIG. 5AL illustrates a transition from FIG. 5AK showing the addition ofa measurement point to user interface 5006. In particular, FIG. 5ALillustrates activation of measurement addition button 5014, while touchinput 5120 is maintained on measurement addition button 5014, by anincrease in the contact intensity of touch input 5120 to above lightpress intensity threshold IT_(L), as indicated by intensity meter 5040.In response to the activation of measurement addition button 5014,device 100 adds and displays (virtual) measurement point 5132 to userinterface 5006 at the current location of focus point 5012 and as thesecond endpoint of measurement segment 5128. In conjunction with addingmeasurement point 5132 to user interface 5006, and in response to theincrease in contact intensity of touch input 5120 above light pressintensity threshold IT_(L), device 100 optionally generates tactileoutput 5134 to indicate the addition of a measurement point. In someembodiments, as shown in FIG. 5AL (and as previously described withreference to FIG. 5AI), the size of measurement addition button 5014 isdecreased as the contact intensity of touch input 5120 increases.Optionally, as shown in FIG. 5AL, upon the completion of measurementsegment 5128, device 100 ceases to display virtual guide 5130.

FIG. 5AM illustrates a transition from FIG. 5AL showing that touch input5120 is maintained on measurement addition button 5014 with a contactintensity that is above minimum contact detection threshold IT₀ but thathas decreased below light press intensity threshold IT_(L). In someembodiments, device 100 generates a tactile output upon detecting thedecrease in the contact intensity of touch input 5120 below light pressintensity threshold IT_(L), as described herein with respect to FIG.5AJ.

FIGS. 5AN-5AR illustrates a transition from FIG. 5AM showing theaddition of another measurement segment that is continuous with (e.g.,has an endpoint in common with) measurement segment 5128, using the samecontinuous touch input 5120. In FIG. 5AN, user 5004 has moved device 100diagonally upward and toward the right while maintaining touch input5120 on measurement addition button 5014 with a contact intensity thatis above minimum contact detection threshold IT₀ and below light pressintensity threshold IT_(L). Device 100 has been moved such that reticle5010 is positioned over an area of the top surface of table 5002 thatdoes not include any features to which focus point 5012 has snapped.Accordingly, focus point 5012 is displayed at the center of reticle5010, and reticle 5010 is displayed at its increased size. Measurementpoint 5122, measurement segment 5128, and measurement point 5132continue to be displayed over the live preview, and dynamic measurementsegment 5136 is displayed between the current position of focus point5012 and measurement point 5122. Because the live preview does notinclude any features that extend in the direction of movement of device100, no virtual guides are displayed. For example, the movement ofdevice 100 is more than a predefined angle from horizontal, so virtualguides are not displayed for features in the live preview that extendhorizontally; in addition, the movement of device 100 is more than apredefined angle from vertical, so virtual guides are not displayed forfeatures in the live preview that extend vertically.

FIG. 5AO illustrates a transition from FIG. 5AN. In FIG. 5AO, user 5004has moved device 100 horizontally toward the right while maintainingtouch input 5120 on measurement addition button 5014 with a contactintensity that is above minimum contact detection threshold IT₀ andbelow light press intensity threshold IT_(L). Device 100 has been movedsuch that measurement point 5122, measurement segment 5128, andmeasurement point 5132 are no longer displayed in user interface 5006because the physical features to which they correspond are no longerwithin the live preview displayed in user interface 5006. Dynamicmeasurement segment 5136 continues to be displayed extending from thecurrent position of focus point 5012. However, because measurement point5132 (the other endpoint of dynamic measurement segment 5136) is nolonger displayed in user interface 5006, dynamic measurement segment5136 extends only to the edge of user interface 5006 (e.g., toward aprojected position of measurement point 5132). In addition, inaccordance with the horizontal movement of device 100, virtual guide5138 is displayed, where virtual guide 5138 extends horizontally from(the projected position of) measurement point 5132 and along thehorizontal front edge of table 5002 in user interface 5006.

FIG. 5AP illustrates a transition from FIG. 5AO. In FIG. 5AP, user 5004has moved device 100 toward the right and slightly downward whilemaintaining touch input 5120 on measurement addition button 5014 with acontact intensity that is above minimum contact detection threshold IT₀and below light press intensity threshold IT_(L). Device 100 has beenmoved such that virtual guide 5138, corresponding to the horizontalfront edge of table 5002 in the live preview, is at least partiallywithin reticle 5010 (e.g., focus point 5012 is within a predefineddistance (e.g., the predefined distance being the radius of reticle5010) of virtual guide 5138 and the horizontal front edge of table 5002in the live preview). Accordingly, focus point 5012 is snapped tovirtual guide 5138 (e.g., to a point along virtual guide 5138 that isthe shortest distance from the center of reticle 5010). Reticle 5010 isdisplayed at its decreased size, and, optionally, tactile output 5140 isgenerated, to indicate the snapping behavior. Dynamic measurementsegment 5136 continues to be displayed extending from the currentposition of focus point 5012 to the edge of user interface 5006 (e.g.,toward a projected position of measurement point 5132). In addition, inaccordance with the continued movement of device 100 that is within apredefined angle of horizontal, horizontal virtual guide 5138 continuesto be displayed along the front edge of table 5002 in user interface5006.

FIG. 5AQ illustrates a transition from FIG. 5AP. In FIG. 5AQ, user 5004has moved device 100 horizontally toward the right while maintainingtouch input 5120 on measurement addition button 5014 with a contactintensity that is above minimum contact detection threshold IT₀ andbelow light press intensity threshold IT_(L). Device 100 has been movedsuch that reticle 5010 is positioned over the upper-front-right cornerof table 5002 as displayed in the live preview. Accordingly, focus point5012 is snapped to the anchor point corresponding to theupper-front-right corner of table 5002 in the live preview. Reticle 5010is displayed at its decreased size to indicate the snapping behavior.Dynamic measurement segment 5136 continues to be displayed extendingfrom the current position of focus point 5012 to the edge of userinterface 5006 (e.g., toward a projected position of measurement point5132). In addition, in the sequence of FIGS. 5AN-5AQ, the labelcorresponding to measurement segment 5136 is updated to reflect thechanges in the length of the measurement represented by dynamicmeasurement segment 5136 as device 100 is moved. In addition, inaccordance with the continued horizontal movement of device 100,horizontal virtual guide 5138 continues to be displayed along the frontedge of table 5002 in user interface 5006.

FIGS. 5AR-5AS illustrate a transition from FIG. 5AQ showing the additionof a measurement point to user interface 5006. In particular, FIG. 5ARillustrates activation of measurement addition button 5014, while touchinput 5120 is maintained on measurement addition button 5014, by anincrease in the contact intensity of touch input 5120 to above lightpress intensity threshold IT_(L), as indicated by intensity meter 5040.In response to the activation of measurement addition button 5014,device 100 adds and displays (virtual) measurement point 5142 to userinterface 5006 at the current location of focus point 5012 and as thesecond endpoint of measurement segment 5136. In conjunction with addingmeasurement point 5142 to user interface 5006, and in response to theincrease in contact intensity of touch input 5120 above light pressintensity threshold IT_(L), device 100 optionally generates tactileoutput 5144 to indicate the addition of a measurement point. Inaddition, the size of measurement addition button 5014 is (optionally)decreased as the contact intensity of touch input 5120 increases.Optionally, as shown in FIG. 5AR, device 100 continues to displayvirtual guide 5138 even after the completion of measurement segment5136.

FIG. 5AS illustrates liftoff of touch input 5120 from measurementaddition button 5014. In some embodiments, after the liftoff of touchinput 5120 from measurement addition button 5014 (and before anysubsequent touch inputs are detected), further movement of device 100will not result in display of a new dynamic measurement segment thatextends from measurement point 5142 (e.g., and that is continuous withmeasurement segment 5136). That is, the liftoff of continuous touchinput 5120 ends the continuous creation of new measurement segmentsbased on touch input 5120.

In some embodiments, the device responds differently to a series of tapgestures (where contact is not maintained with the touch-sensitivesurface) than to a series of pressing gestures (where contact ismaintained with the touch-sensitive surface). For inputs that arediscrete tap gestures (rather than pressing gestures made with a singlecontinuously detected contact), measurement segments are notcontinuously created with each subsequently added measurement point.That is, for a series of tap gestures, if a user drops four points insuccession, a measurement segment will be created between the firstpoint and second point, and another measurement segment will be createdbetween the third point and the fourth point, but a measurement segmentwill not be created between the second point and the third point.

FIG. 5AT illustrates a transition from FIG. 5AS. In FIG. 5AT, device 100has been moved such that reticle 5010 is positioned over theupper-front-left corner of table 5002 as displayed in the live previewand over measurement point 5132. Accordingly, focus point 5012 issnapped to measurement point 5132, at the anchor point corresponding tothe upper-front-left corner of table 5002 in the live preview. Reticle5010 is displayed at its decreased size to indicate the snappingbehavior. Measurement segment 5128 and measurement segment 5136 aredisplayed in user interface 5006 at their respective locationscorresponding to respective features of table 5002 (e.g., the horizontalupper-left edge and the horizontal front edge, respectively, of table5002), in accordance with the corresponding features of table 5002 beingdisplayed (again) in the live preview in user interface 5006.Measurement segment 5128 is displayed between its endpoints, measurementpoint 5122 and measurement point 5132. Measurement segment 5136 isdisplayed extending from measurement point 5132 (one endpoint ofmeasurement segment 5136) to the edge of user interface 5006 (e.g.,toward a projected position of measurement point 5142, which is theother endpoint of measurement segment 5136, and which is not currentlyvisible in user interface 5006). In addition, in accordance with reticle5010 being positioned over measurement point 5132, virtual guides 5146,5148, and 5150 extending from measurement point 5132 are displayed.Virtual guide 5146 extends in an x-direction (e.g., horizontally alongthe horizontal front edge of table 5002) from measurement point 5132.Virtual guide 5148 extends in a y-direction (e.g., vertically along thevertical front-left edge of table 5002) from measurement point 5132.Virtual guide 5150 extends in a z-direction (e.g., horizontally alongthe horizontal upper-left edge of table 5002) from measurement point5132.

FIGS. 5AU-5AX illustrate a transition from FIG. 5AT showing the additionof a measurement segment to user interface 5006. FIG. 5AU illustratesactivation of measurement addition button 5014 by touch input 5152(e.g., a tap gesture) with a contact intensity that is above a minimumcontact detection threshold IT₀, as indicated by intensity meter 5040.In response to the activation of measurement addition button 5014,device 100 uses measurement point 5132 as the starting point for a newmeasurement segment to be added and, optionally, generates tactileoutput 5154 to indicate the beginning of a new measurement segment.

FIG. 5AV illustrates that (after liftoff of touch input 5152) user 5004has moved device 100 downward along the vertical front-left edge oftable 5002. Focus point 5012 is snapped to an anchor point along thevertical front-left edge of table 5002 that is not the lower-front-leftcorner of table 5002. Reticle 5010 is displayed at its decreased size toindicate the snapping behavior. Also, in accordance with the verticalmovement, device 100 continues to display virtual guide 5148, whichextends vertically from measurement point 5132, and has ceased todisplay virtual guides 5146 and 5150, which do not extend vertically (orin a direction that is within a predefined angle of vertical) frommeasurement point 5132. In addition, dynamic measurement segment 5156 isdisplayed between measurement point 5132 and the current position offocus point 5012.

FIG. 5AW illustrates activation of measurement addition button 5014 bytouch input 5158 (e.g., a tap gesture) with a contact intensity that isabove a minimum contact detection threshold IT₀, as indicated byintensity meter 5040. In response to the activation of measurementaddition button 5014, device 100 adds and displays measurement point5160 to user interface 5006 at the current location of focus point 5012and as the second endpoint of measurement segment 5156, which becomes acompleted measurement segment. In conjunction with adding measurementpoint 5160 to user interface 5006, device 100 optionally generatestactile output 5162 (e.g., to indicate the completion of a measurementsegment). FIG. 5AX illustrates liftoff of touch input 5158 frommeasurement addition button 5014.

FIGS. 5AY-5BE illustrate example zoom interactions with the augmentedreality environment in user interface 5006. In particular, FIG. 5AY-5BAillustrate zoom-assisted repositioning of a displayed measurement point.FIG. 5AY illustrates touch input 5164 detected on measurement point 5160with a contact intensity that is above a minimum contact detectionthreshold IT₀, as indicated by intensity meter 5040. In response todetecting touch input 5164 on measurement point 5160, device 100enlarges, or zooms into, a portion of the live preview that includesmeasurement point 5160 (e.g., the portion of the live preview that iscentered on measurement point 5160). An amount of zoom of the livepreview is based on the distance between device 100 and the point ontable 5002 to which measurement point 5160 corresponds (e.g., a point ontable 5002 just above the lower-front-left corner of table 5002). Forexample, in FIG. 5AY, when device 100 is a distance d₁ from the point ontable 5002 to which measurement point 5160 corresponds, the live previewis enlarged by a zoom factor of 4×.

FIG. 5AZ illustrates a transition from FIG. 5AY showing movement oftouch input 5164 across touch screen 112 (e.g., a pan gesture or a draggesture by the contact in touch input 5164) such that touch input 5164is over an anchor point corresponding to the lower-front-left corner oftable 5002. Measurement point 5160 moves in user interface 5006 with themovement of touch input 5164 across touch screen 112. In someembodiments, as shown in FIG. 5AZ, measurement point 5160 is snapped tothe anchor point over which touch input 5164 has moved. Accordingly,measurement point 5160 is displayed at the anchor point corresponding tothe lower-front-left corner of table 5002. In conjunction with themovement of measurement point 5160, measurement segment 5156 is extendedand its label is updated accordingly (to indicate a length of thevertical front-left edge of table 5002). In addition, device 100optionally generates tactile output 5166 to indicate the snappingbehavior.

In some embodiments, device 100 determines a vector from the position ofthe camera to the location on a detected surface in physical space 5000over which a measurement point is displayed. In some embodiments, device100 determines an angle between the determined vector and the detectedsurface. In some embodiments, in accordance with a determination thatthe determined vector is within a predefined threshold angle of thedetected surface (e.g., the determined angle is less than a predefinedthreshold angle, such as 15, 18, 20, 25, or 30 degrees), when receivinga set of one or more user inputs to move the measurement point, themeasurement point is moved through locations in user interface 5006 thatcorrespond to locations along the determined vector.

FIG. 5BA illustrates a transition from FIG. 5AZ. In some embodiments, asshown in FIG. 5BA, upon liftoff of touch input 5164, device 100 ceasesto display the enlarged (portion of the) live preview and redisplays thelive preview without zoom. In some embodiments, after liftoff of touchinput 5164, device 100 continues to display the enlarged live previewuntil a subsequent input (to exit the zoomed live preview and return tothe live preview displayed without zoom) is detected.

Because device 100 was maintained in the same position in FIGS. 5AX-5AZ,reticle 5010 is displayed in FIG. 5BA at the same size and at the sameposition as it was in FIG. 5AX (prior to the zoom-assisted repositioningof measurement point 5160). Because measurement point 5160 wasrepositioned to the anchor point corresponding to the lower-front-leftcorner of table 5002 as described with respect to FIGS. 5AY-5AZ,measurement point 5160 is displayed at that anchor point in the livepreview that is displayed without zoom in FIG. 5BA, outside of reticle5010. Extended measurement segment 5156 and its corresponding updatedlabel are also displayed.

FIG. 5BB-5BC illustrate another example zoom interaction. FIG. 5BB issimilar to FIG. 5AX, except that device 100 is positioned closer totable 5002 (or, more specifically, to the point on table 5002 to whichmeasurement point 5160 corresponds) in FIG. 5BB than in FIG. 5AX (asindicated by the side view of user 5004, device 100, and table 5002 inFIG. 5AY), at a distance d2 that is less than the distance d₁ in FIG.5AY. Accordingly, in response to detecting touch input 5168 onmeasurement point 5160, as shown in FIG. 5BC, device 100 zooms into aportion of the live preview that includes measurement point 5160. Theamount of zoom of the live preview in FIG. 5BC is based on the lesserdistance d2 between device 100 and table 5002, and thus the amount ofzoom of the live preview in FIG. 5BC, corresponding to a zoom factor of2×, is less than the amount of zoom in FIG. 5AY, which corresponds to azoom factor of 4×. In addition, because device 100 is closer to table5002 in FIG. 5BB than in FIG. 5AX, scale markers are displayed atone-foot intervals along measurement segment 5156 (e.g., as opposed tono scale markers being displayed in FIG. 5AX). Also, because device 100is closer to table 5002 in FIG. 5BB than in FIG. 5AX, the size of thelabels corresponding to the displayed measurements is larger in FIG. 5BBthan in FIG. 5AX.

FIG. 5BD-5BE illustrate another example zoom interaction. FIG. 5BD issimilar to FIG. 5BB, except that device 100 is positioned closer totable 5002 (or, more specifically, to the point on table 5002 to whichmeasurement point 5160 corresponds) in FIG. 5BD than in FIG. 5BB, at adistance d₃ that is less than the distance d2 in FIG. 5BB. Accordingly,in response to detecting touch input 5170 on measurement point 5160, asshown in FIG. 5BE, device 100 zooms into a portion of the live previewthat includes measurement point 5160. The amount of zoom of the livepreview in FIG. 5BE is based on the lesser distance d₃ between device100 and table 5002, and thus the amount of zoom of the live preview inFIG. 5BE, corresponding to a zoom factor of 1.5×, is less than theamount of zoom in FIG. 5BC, which corresponds to a zoom factor of 2×. Inaddition, because device 100 is closer to table 5002 in FIG. 5BD than inFIG. 5BD, scale markers are displayed at one-inch intervals alongmeasurement segment 5156 (e.g., as opposed to the scale markersdisplayed at one-foot intervals in FIG. 5BB). Also, because device 100is closer to table 5002 in FIG. 5BD than in FIG. 5BB, the size of thelabels corresponding to the displayed measurements is larger in FIG. 5BDthan in FIG. 5BB. In some embodiments, the scale at which markers aredisplayed along a measurement segment becomes finer as the distancebetween device 100 and the physical feature corresponding to themeasurement segment decreases (e.g., at distances above a firstdistance, no scale markers are displayed; at distances between the firstdistance and a second distance (shorter than the first distance), scalemarkers are displayed at one-foot intervals; at distances between thesecond distance and a third distance (shorter than the second distance),scale markers are displayed at one-inch intervals; at distances shorterthan the third distance, scale markers are displayed at quarter-inchintervals, and so on).

In some embodiments, the amount of zoom displayed in FIG. 5AY is amaximum amount of zoom, such that when the distance between device 100and table 5002 (or the point on table 5002 to which the displayedmeasurement point corresponds) is greater than the distance d₁ shown inFIG. 5AY, the amount of zoom of the live preview still corresponds to azoom factor of 4×. In some embodiments, the amount of zoom displayed inFIG. 5BE is a minimum amount of zoom, such that when the distancebetween device 100 and table 5002 (or the point on table 5002 to whichthe displayed measurement point corresponds) is less than the distanced₃ shown in FIG. 5BE, the amount of zoom of the live preview stillcorresponds to a zoom factor of 1.5×.

Similarly, in some embodiments, the size of the labels displayed in FIG.5AY is a minimum label size, such that when the distance between device100 and table 5002 (or the point on table 5002 to which the displayedmeasurement point corresponds) is greater than the distance d₁ shown inFIG. 5AY, the size of the labels is the same as in FIG. 5AY. In someembodiments, the size of the labels displayed in FIG. 5BE is a maximumlabel size, such that when the distance between device 100 and table5002 (or the point on table 5002 to which the displayed measurementpoint corresponds) is less than the distance d₃ shown in FIG. 5BE, thesize of the labels is the same as in FIG. 5BE.

FIG. 5BF-5BK illustrate capturing images of the augmented realityenvironment in user interface 5006. FIG. 5BF optionally illustrates atransition from FIG. 5BA. In FIG. 5BF, user 5004 has positioned device100 such that measurement segments 5128, 5136, and 5156, and theircorresponding endpoints and labels, are displayed in user interface5006, the corresponding features of table 5002 being visible in the livepreview. FIG. 5BG illustrates activation of media capture button 5016 bytouch input 5172, as indicated by the increase in intensity shown inintensity graph 5180, which shows the contact intensity of touch input5172 over time.

FIG. 5BH illustrates a transition from FIG. 5BG based on liftoff oftouch input 5172 before a predefined threshold time Tin (e.g., touchinput 5172 is a tap gesture). Accordingly, intensity graph 5180 shows acorresponding decrease in the contact intensity of touch input 5172 tozero before time Tin. In response to detecting liftoff of touch input5172 before the predefined threshold time Tin, device 100 captures image5174 of the augmented reality environment. Captured image 5174 is astill image that includes an image of the field of view of the camera,corresponding to an instantaneous snapshot of the live preview, andmeasurement segments 5128, 5136, and 5156, along with theircorresponding endpoints and labels, superimposed on the image of thefield of view of the camera. In some embodiments, as shown in FIG. 5BH,captured image 5174 does not include images of the buttons/controls inuser interface 5006.

FIG. 5BI illustrates capture of an image in response to activation ofmedia capture button 5016 while device 100 is at a different positionrelative to table 5002 (facing the left side surface of table 5002) suchthat a different perspective view of table 5002 is displayed in the livepreview. In response to activation of media capture button 5016 by atouch input and liftoff of the touch input before the predefinedthreshold time T_(th), as indicated by intensity graph 5180, device 100captures image 5176 of the augmented reality environment. Captured image5176 is a still image that includes an image of table 5002 from theperspective of device 100 at its position in FIG. 5BI (e.g., facing theleft side surface of table 5002). Measurement segments 5128, 5136, and5156 and their corresponding endpoints and labels are superimposed onthe corresponding features of table 5002 in captured image 5176 based onthe perspective of device 100 in FIG. 5BI.

FIG. 5BJ illustrates an alternate transition from FIG. 5BG. Togetherwith FIG. 5BG, FIGS. 5BJ-5BK illustrate capture of a video of theaugmented reality environment in response to a touch input maintained onmedia capture button 5016 (e.g., a long-press gesture). In FIG. 5BJ,touch input 5172 is maintained on media capture button 5016 past thepredefined threshold time T_(th), as indicated by intensity graph 5180.Accordingly, device 100 captures video of the field of view of thecamera. Timer 5178 is displayed in user interface 5006 and indicates acurrent length of the captured video. In addition, the captured videoincludes any measurements in the field of view of the camera (e.g.,measurement segments 5128, 5136, and 5156, and their correspondingendpoints and labels, are superimposed on the corresponding features oftable 5002 in the captured video). In some embodiments, the capturedvideo does not include images of the buttons/controls in user interface5006.

FIG. 5BK illustrates that device 100 has moved while touch input 5172 ismaintained on media capture button 5016. Accordingly, device 100 hascontinued to capture video of the field of view of the camera as device100 moved, as indicated by the current video length shown by timer 5178in FIG. 5BK being longer than that shown by timer 5178 in FIG. 5BJ. Thecaptured video includes additional portions of measurement segment 5136superimposed over corresponding features in the field of view of thecamera as device 100 moves to its position as shown in FIG. 5BK.

FIGS. 5BL-5BM illustrate displaying additional information about aselected measurement and options for sharing the information to anotherapplication, process, or device. FIG. 5BL illustrates touch input 5182(e.g., a tap gesture by a contact in the touch input) detected onmeasurement segment 5156 with a contact intensity that is above aminimum contact detection threshold IT₀, as indicated by intensity meter5040. FIG. 5BM illustrates that, in response to detecting touch input5182 on measurement segment 5156, device 100 displays measurementmanagement interface 5184.

Measurement management interface 5184 includes a label that describesthe physical object, table 5002, to which measurement 5156 corresponds.For example, measurement management interface 5184 includes label 5186-aclassifying table 5002 (e.g., identifying table 5002 as a “Table”).Measurement management interface 5184 also includes label 5186-bclassifying the relationship between measurement 5156 and table 5002(e.g., identifying measurement 5156 as a “height” of table 5002). Insome embodiments, the relationship between measurement 5156 and table5002 is classified as a “height” based on the vertical movement ofdevice 100 while adding measurement 5156 to user interface 5006.

In some embodiments, in response to touch input 5182, information aboutmeasurement 5156 (e.g., the classification of the physical object towhich measurement 5156 corresponds and the relationship betweenmeasurement 5156 and the physical object, a magnitude of measurement5156 such as length or area, an image of measurement 5156, etc.) iscopied to a clipboard process executing on device 100. In someembodiments, measurement management interface includes a plurality ofdestinations to which information about selected measurement 5156 can betransmitted (e.g., icon 5192 corresponding to e-mail client module 140(FIG. 1A), icon 5194 corresponding to IM module 141 (FIG. 1A), and icon5196 corresponding to a file transfer protocol between electronicdevices). In some embodiments, measurement management interface 5184 isdisplayed in response to touch input 5182 satisfying an intensitythreshold (e.g., light press intensity threshold IT_(L)) that is aboveminimum contact detection threshold IT₀ (e.g., in response to touchinput 5182 being a light press or deep press gesture).

FIG. 5BN illustrates an example control center user interface 5188 thatincludes augmented reality measurement application icon 5190. Activationof augmented reality measurement application icon 5190 launches theaugmented reality measurement application and displays user interface5006 (e.g., as described with reference to FIG. 5A).

FIG. 5BO illustrates a context in which user interfaces described withrespect to FIGS. 5BO-5CO are used. FIG. 5BO is similar to FIG. 5A inthat FIG. 5BO illustrates a view of physical space 5000 that includesdevice 100, except that physical space 5000 includes table 5200 on whichobject 5202 is placed (instead of table 5002). Object 5202 is in thefield of view of the camera(s) of device 100 and is visible in the livepreview of physical space 5000 displayed in user interface 5006 ondevice 100. In some embodiments, as shown in FIG. 5BO, reticle 5010 inuser interface 5006 is tilted to appear to be co-planar with the topsurface of table 5200 to indicate the surface that has been detected andthat corresponds to the current location of focus point 5012.

FIGS. 5BP-5BQ illustrate a first way of adding a virtual measurementpoint to user interface 5006, in accordance with some embodiments. FIG.5BP shows touch input 5204 on reticle 5010. In accordance with someembodiments, FIG. 5BP shows that, in response to touch input 5204,device 100 adds and displays virtual measurement point 5206 to userinterface 5006 at a current location of focus point 5012 (as shown inFIG. 5BO). FIG. 5BQ shows measurement point 5206 displayed at the samelocation as in FIG. 5BP after liftoff of touch input 5204.

FIGS. 5BR-5BS illustrate an alternate way of adding a virtualmeasurement point to user interface 5006, in accordance with someembodiments. FIG. 5BR shows touch input 5204 on reticle 5010. Incontrast to FIG. 5BP, FIG. 5BR shows that, in response to touch input5204, device 100 forgoes adding and displaying a virtual measurementpoint at the location of focus point 5012. Instead, in FIG. 5BR, device100 displays instruction message 5208 to prompt user 5004 to tap onmeasurement addition button 5014 (instead of tapping on reticle 5010) toadd a measurement point. FIG. 5BS illustrates activation of measurementaddition button 5014 by touch input 5210. In response to touch input5210, device 100 adds virtual measurement point 5206 to user interface5006 at a current location of focus point 5012 (as shown in FIG. 5BQ).

FIG. 5BT-5BU illustrate the creation of a measurement corresponding toobject 5202 following the addition of measurement point 5206 (e.g., ineither FIG. 5BQ or FIG. 5BS). In FIG. 5BT, user 5004 has moved device100 such that reticle 5010 and focus point 5012 are positioned over adifferent location in physical space 5000. Specifically, in FIG. 5BT,reticle 5010 and focus point 5012 are positioned over an edge of object5202 (as displayed in the live preview) that is closer to device 100than the edge over which reticle 5010 and focus point 5012 werepositioned in FIG. 5BO. Accordingly, reticle 5010 is displayed at anincreased size in FIG. 5BT relative to its size in FIG. 5BO, and focuspoint 5012 is displayed at an increased size in FIG. 5BT relative to itssize in FIG. 5BO. In some embodiments, the size at which reticle 5010 isdisplayed is based on a distance between device 100 and the location inphysical space 5000 over which reticle 5010 is displayed, optionallysubject to a predefined minimum size (used for distances greater than apredefined maximum distance) and a predefined maximum size (used fordistances less than a predefined minimum distance). Similarly, in someembodiments, the size at which focus point 5012 is displayed is based ona distance between device 100 and the location in physical space 5000over which focus point 5012 is displayed, optionally subject to apredefined minimum size and a predefined maximum size. In addition, inaccordance with the movement of device 100 such that reticle 5010 andfocus point 5012 are positioned over a different location in physicalspace 5000, (dynamic) measurement segment 5212, indicated by a dashedline, is displayed between measurement point 5206 (themost-recently-added measurement point) and focus point 5012. Measurementsegment 5212 is displayed with an associated (dynamic) label thatindicates a distance in physical space 5000 across which measurementsegment 5212 appears to extend in user interface 5006.

FIG. 5BU illustrates activation of measurement addition button 5014 bytouch input 5214 (e.g., a tap gesture). In response to the activation ofmeasurement addition button 5214, device 100 adds and displaysmeasurement point 5216 in user interface 5006 at a current location offocus point 5012 (as shown in FIG. 5BT) and as the second endpoint ofmeasurement segment 5212. In accordance with the completion ofmeasurement segment 5212, the appearance of measurement segment 5212 ischanged. In the example shown in FIG. 5BU, measurement segment 5212 ischanged from a dashed line to a solid line. Because measurement point5216 is positioned over an edge of object 5202 that is closer to device100 than the edge of object 5202 over which measurement point 5206 isdisplayed, measurement point 5216 is displayed at an increased sizerelative to measurement point 5206.

FIG. 5BV-5BY illustrate creation of a measurement corresponding toobject 5202 that causes a prior measurement to be removed, in accordancewith some embodiments. FIG. 5BV illustrates a transition from FIG. 5BU.In FIG. 5BV, user 5004 has moved device 100 such that reticle 5010 andfocus point 5012 are positioned over a different location in physicalspace 5000 than in FIG. 5BU. Specifically, in FIG. 5BV, reticle 5010 andfocus point 5012 are positioned over a first corner of object 5202. Insome embodiments, as shown in FIG. 5BV, in accordance with reticle 5010and focus point 5012 being moved away from measurement segment 5212,device 100 ceases to display the label (“17 in”) associated withmeasurement segment 5212.

FIG. 5BW illustrates a transition from FIG. 5BV showing the addition ofa measurement point to user interface 5006. In particular, FIG. 5BWillustrates activation of measurement addition button 5014 by touchinput 5218. In response, device 100 adds measurement point 5220 to userinterface 5006 at the current location of focus point 5012 (as shown inFIG. 5BV). In addition, in accordance with the addition of a newmeasurement point away from previously-created measurement segment 5212,device 100 changes the appearance of measurement segment 5212 (e.g., toindicate that creating a measurement segment that is disconnected frommeasurement segment 5212 will cause measurement segment 5212 to beremoved from user interface 5006). In the example shown in FIG. 5BW,measurement segment 5212 is changed from a solid line to a dashed line,and the color (and/or transparency) of measurement segment 5212 and itsendpoints is changed.

FIG. 5BX illustrates a transition from FIG. 5BW showing that user 5004has moved device 100 such that reticle 5010 and focus point 5012 arepositioned over a second corner of object 5202. Accordingly, (dynamic)measurement segment 5222, indicated by a dashed line, is displayedbetween measurement point 5206 (the most-recently-added measurementpoint) and focus point 5012. Measurement segment 5222 is displayed withan associated (dynamic) label (e.g., “17 in”) that indicates a distancein physical space 5000 (e.g., along object 5202) across whichmeasurement segment 5222 appears to extend in user interface 5006.

FIG. 5BY illustrates a transition from FIG. 5BX showing the addition ofmeasurement point 5224 to user interface 5006 in response to activationof measurement addition button 5014 by touch input 5226. FIG. 5BYillustrates that measurement point 5224 has been added at a currentlocation of focus point 5012 (as shown in Figure BX) as the secondendpoint of measurement segment 5222. Because measurement point 5224 ispositioned over a location on object 5202 that is further from device100 than the location on object 5202 over which measurement point 5220is displayed, measurement point 5224 is displayed at a decreased sizerelative to measurement point 5220. In accordance with the completion ofmeasurement segment 5222, the appearance of measurement segment 5222 ischanged from a dashed line to a solid line. In addition, in accordancewith the completion of measurement segment 5222, and in accordance withmeasurement segment 5222 being disconnected from previously-placedmeasurement segment 5212, device 100 ceases to display measurementsegment 5212. In some embodiments, measurement segment 5212 ceases to bedisplayed in accordance with a determination that measurement segment5222 is at least a predefined threshold distance away from measurementsegment 5212 (e.g., no point on measurement segment 5212 is within thepredefined threshold distance of any point on measurement segment 5222).

FIGS. 5BZ-5CF illustrate creation of a measurement corresponding toobject 5202 that connects to a prior measurement such that the priormeasurement continues to be displayed, in accordance with someembodiments. FIG. 5BZ illustrates a transition from FIG. 5BY showingthat user 5004 has moved device 100 such that reticle 5010 and focuspoint 5012 are positioned over a third corner of object 5202. In someembodiments, as shown in FIG. 5BZ, even though reticle 5010 and focuspoint 5012 have been moved away from measurement segment 5222, device100 continues to display the label associated with measurement segment5222 (in contrast to FIG. 5BV, which illustrates embodiments in whichdevice 100 ceases to display the label associated with measurementsegment 5212 when reticle 5010 and focus point 5012 are moved away fromthe measurement segment).

FIG. 5CA illustrates a transition from FIG. 5BZ showing the addition ofmeasurement point 5228 to user interface 5006 at a current location offocus point 5012 (as shown in FIG. 5BZ) in response to activation ofmeasurement addition button 5014 by touch input 5230. In accordance withmeasurement point 5228 being added at a location in user interface 5006that is away from previously-created measurement segment 5222, device100 changes the appearance of measurement segment 5222 (e.g., toindicate that creating a measurement segment that is disconnected frommeasurement segment 5222 will cause measurement segment 5222 to beremoved from user interface 5006). In the example shown in FIG. 5CA,measurement segment 5222 is changed from a solid line to a dashed line,and the color (and/or transparency) of measurement segment 5222 ischanged.

FIG. 5CB illustrates a transition from FIG. 5BZ showing that user 5004has moved device 100 such that reticle 5010 and focus point 5012 arepositioned away from the third corner of object 5202. Accordingly,(dynamic) measurement segment 5232, indicated by a dashed line, isdisplayed between measurement point 5228 and focus point 5012.Measurement segment 5232 is displayed with an associated (dynamic) labelthat indicates the distance along object 5202 across which measurementsegment 5222 appears to extend in user interface 5006 (e.g., “8 in”).

FIG. 5CC illustrates a transition from FIG. 5CB showing that user 5004has moved device 100 such that the midpoint of measurement segment 5222is within reticle 5010. In some embodiments, as shown in FIG. 5CC, themidpoint of a respective measurement segment displayed in user interface5006 can be an anchor point to which focus point 5012 snaps.Accordingly, in FIG. 5CC, focus point 5012 is snapped to the anchorpoint corresponding to the midpoint of measurement segment 5222. Toindicate the snapping behavior, focus point 5012 is displayed at anincreased size relative to the size of focus point 5012 when focus point5012 is not snapped to an anchor point (e.g., as shown in FIG. 5CB). Insome embodiments, as shown in FIG. 5CC, the size of reticle 5010 is notchanged when the focus point is snapped to an anchor point. In someembodiments, the size of focus point 5012 when snapped to an anchorpoint is greater than the predefined maximum size of focus point 5012that is used for changing the size of focus point 5012 based on thedistance between device 100 and the location in physical space 5000 overwhich focus point 5012 is displayed (e.g., as described herein withrespect to FIG. 5BT). In addition, because focus point 5012 is snappedto a point along measurement segment 5222, the appearance of measurementsegment 5222 is changed to indicate that adding a measurement point atthe current location of focus point 5012 will result in measurementsegment 5222 continuing to be displayed instead of being removed.Specifically, measurement segment 5222 is changed from a dashed line(back) to a solid line, and the color (and/or transparency) ofmeasurement segment 5222 is changed, such that measurement segment 5222is redisplayed with its appearance as shown in FIG. 5BZ (beforemeasurement point 5228 was added). In addition, the length of dynamicmeasurement segment 5232 is updated in accordance with the movement ofdevice 100 such that measurement segment 5232 continues to be displayedbetween measurement point 5228 and the current location of focus point5012. The label associated with measurement segment 5232 is updated toreflect the change in length of measurement segment 5232 (e.g., “12in”).

FIG. 5CD illustrates a transition from FIG. 5CC showing that user 5004has moved device 100 such that reticle 5010 and focus point 5012 arepositioned away from the midpoint of measurement segment 5222.Accordingly, in FIG. 5CD, dynamic measurement segment 5232 and itsassociated label are updated to reflect the change in length ofmeasurement segment 5232 due to the movement of focus point 5012relative to measurement point 5228. In addition, the appearance ofmeasurement segment 5222 is updated to indicate that adding ameasurement point at the current location of focus point 5012 willresult in measurement segment 5222 being removed.

FIG. 5CE illustrates a transition from FIG. 5CD showing that user 5004has moved device 100 such that measurement point 5224 (as shown in FIG.5CD) is within reticle 5010. In some embodiments, as shown in FIG. 5CE,the endpoint of a previously-added measurement segment can be an anchorpoint to which focus point 5012 snaps. Accordingly, in FIG. 5CE, focuspoint 5012 is snapped to the anchor point corresponding to measurementpoint 5224 (as shown in FIG. 5CD). To indicate the snapping behavior,focus point 5012 is displayed at an increased size relative to its sizewhen not snapped to an anchor point (e.g., as shown in FIG. 5CD),although the size of reticle 5010 is not changed, as described hereinwith reference to FIG. 5CC. In addition, because focus point 5012 issnapped to a point along measurement segment 5222, the appearance ofmeasurement segment 5222 is changed to indicate that adding ameasurement point at the current location of focus point 5012 willresult in measurement segment 5222 continuing to be displayed, insteadof being removed. Specifically, measurement segment 5222 is changed froma dashed line to a solid line, and the color (and/or transparency) ofmeasurement segment 5222 is changed, such that measurement segment 5222is redisplayed with its appearance as shown in FIG. 5BZ. In addition,dynamic measurement segment 5232 and its associated label are updated toreflect the change in length of measurement segment 5232 due to themovement of focus point 5012 relative to measurement point 5228.

FIG. 5CF illustrates a transition from FIG. 5CE showing the addition ofmeasurement point 5234 to user interface 5006 at the current location offocus point 5012 (as shown in FIG. 5CE) in response to activation ofmeasurement addition button 5014 by touch input 5236. In accordance withmeasurement point 5234 being added at a point along measurement segment5222, measurement segment 5222 continues to be displayed. In accordancewith the completion of measurement segment 5232, the appearance ofmeasurement segment 5232 is changed from a dashed line to a solid line.

FIG. 5CG-5CK illustrate creation of a measurement that is close enoughto (e.g., within a predefined threshold distance of, but not connectedto) a prior measurement such that the prior measurement continues to bedisplayed, in accordance with some embodiments. FIG. 5CG illustrates atransition from FIG. 5CF showing that user 5004 has moved device 100such that reticle 5010 and focus point 5012 are positioned away fromobject 5202 and over a first corner of table 5200. In some embodiments,as shown in FIG. 5CG, even though reticle 5010 and focus point 5012 havebeen moved away from measurement segment 5222 and measurement segment5232, device 100 continues to display the labels associated withmeasurement segments 5222 and 5232 (in contrast to FIG. 5BV, whichillustrates embodiments in which device 100 ceases to display the labelassociated with measurement segment 5212 when reticle 5010 and focuspoint 5012 are moved away from the measurement segment).

FIG. 5CH illustrates a transition from FIG. 5CG showing the addition ofmeasurement point 5238 to user interface 5006 at the current location offocus point 5012 (as shown in FIG. 5CH) in response to activation ofmeasurement addition button 5014 by touch input 5240. In accordance withmeasurement point 5238 being added at a location in user interface 5006that is away from previously-created measurement segments 5222 and 5232,device 100 changes the appearance of measurement segments 5222 and 5232to indicate that creating a measurement segment that is disconnected andmore than a threshold distance from any point along measurement segment5222 and any point along measurement segment 5232 will cause measurementsegments 5222 and 5232 to be removed from user interface 5006. In theexample shown in FIG. 5CH, measurement segments 5222 and 5232 arechanged from solid lines to dashed lines, and the color (and/ortransparency) of measurement segments 5222 and 5232 and their endpointsis changed.

FIG. 5CI illustrates a transition from FIG. 5CH showing that user 5004has moved device 100 such that reticle 5010 and focus point 5012 arepositioned over a side surface of table 5200. In some embodiments, asshown in FIG. 5CI, reticle 5010 is tilted to appear to be co-planar withthe side surface of table 5200 to indicate the surface that has beendetected and that corresponds to the current location of focus point5012. Dynamic measurement segment 5242, indicated by a dashed line, isdisplayed between measurement point 5238 and focus point 5012, alongwith an associated dynamic label that indicates the distance along theside surface of table 5200 across which measurement segment 5242 appearsto extend in user interface 5006 (e.g., “2 ft 10 in”).

FIG. 5CJ illustrates a transition from FIG. 5CI showing that user 5004has moved device 100 such that reticle 5010 and focus point 5012 arepositioned over a second corner of table 5200. In some embodiments, asshown in FIG. 5CJ, the appearance of measurement segments 5222 and 5232are not changed even though adding a measurement point at the currentlocation of focus point 5012 will result in measurement segments 5222and 5232 continuing to be displayed instead of being removed (e.g.,because the resulting measurement segment that would be created by theaddition of a measurement point at the current location of focus point5012 will be within a predefined threshold distance of both measurementsegments 5222 and 5232).

FIG. 5CK illustrates a transition from FIG. 5CJ showing the addition ofmeasurement point 5244 at the current location of focus point 5012 (asshown in FIG. 5CJ) in response to activation of measurement additionbutton 5014 by touch input 5246. Measurement point 5244 is added at alocation in user interface 5006 such that measurement segment 5242 iswithin a predefined threshold distance of measurement segment 5222(e.g., measurement segment 5242 includes at least one point that iswithin a predefined threshold distance of at least one point alongmeasurement segment 5222). Accordingly, measurement segment 5222continues to be displayed after the addition of measurement point 5244.In addition, measurement segment 5222 is changed from a dashed line(back) to a solid line, and the color (and/or transparency) ofmeasurement segment 5222 is changed, such that measurement segment 5222is redisplayed with its appearance as shown in FIG. 5CG. Similarly,measurement segment 5242 is within the predefined threshold distance ofmeasurement segment 5232 (e.g., measurement segment 5242 includes atleast one point that is within a predefined threshold distance of atleast one point along measurement segment 5222, such as endpointmeasurement point 5234). Accordingly, like measurement segment 5222,measurement segment 5232 continues to be displayed and is redisplayedwith its appearance as shown in FIG. 5CG. In some embodiments, as longas at least one point in any of the currently displayed segments (e.g.,previously connected segments 5222 and 5232) is within the predefineddistance of a newly created segment (e.g., segment 5242), then all ofthe currently displayed segments remain displayed.

FIG. 5CL-5CM illustrate an example alert condition in the augmentedreality environment. FIG. 5CL illustrates a transition from FIG. 5CKshowing that user 5004 has moved device 100 such that field of view ofthe camera no longer includes the portion of physical space 5000 overwhich measurement segments 5222, 5232, and 5242 were displayed (e.g.,the left portion of table 5200, as shown in FIG. 5CK). FIG. 5CLindicates that the amount of time that has elapsed since the field ofview of the camera was moved away from the left portion of table 5200 isless than a first predefined threshold amount of time T_(th).

FIG. 5CM illustrates a transition from FIG. 5CL showing that the amountof time that has elapsed since the field of view of the camera was movedaway from the left portion of table 5200 has reached the firstpredefined threshold amount of time. Accordingly, device 100 displaysalert message 5248 to indicate that measurement segments 5222, 5232, and5242 will soon be removed from user interface 5006. In some embodiments,if device 100 is moved back to its position as shown in FIG. 5CK withina second predefined threshold amount of time since alert message 5248was displayed, measurement segments 5222, 5232, and 5242 will beredisplayed in user interface 5006 over the corresponding features inphysical space 5000 as shown in FIG. 5CG. In some embodiments, if device100 is moved back to its position as shown in FIG. 5CK after the secondpredefined threshold amount of time has elapsed since alert message 5248was displayed, measurement segments 5222, 5232, 5242 will not beredisplayed in user interface 5006. In some embodiments, alert message5248 is displayed when device 100 has been moved such that the portionof physical space 5000 over which measurements 5222, 5232, and 5242 weredisplayed is more than a threshold distance from the portion of physicalspace 5000 that is currently in the field of view of the camera.

FIG. 5CN-5CO illustrate another example alert condition in the augmentedreality environment. FIG. 5CN shows device 100 positioned at a firstdistance from table 5200, where the first distance is less than apredefined (maximum) threshold distance an. Reticle 5010 and focus point5012 are displayed, which indicates that device 100 has detected asurface at the location over which focus point 5012 is displayed. FIG.5CO shows device 100 positioned at a second distance from table 5200,where the second distance is greater than the predefined (maximum)threshold distance an. Accordingly, device 100 does not display reticle5010 and focus point 5012, which indicates that device 100 has notdetected a surface at the location in physical space 5000 over whichfocus point 5012 would have been displayed. In addition, device 100displays alert message 5250 to indicate that device 100 is too far awayfrom the location in physical space 5000 over which focus point 5012would have been displayed and to prompt user 5004 to move device 100closer to that location. Similarly, in some embodiments, when device 100is positioned at a distance from table 5200 that is less than apredefined minimum threshold distance, device 100 does not displayreticle 5010 and focus point 5012 (to indicate that a surface has notbeen detected) and displays an alert message (e.g., with text such as“Move further”) to prompt user 5004 to move device 100 further away fromthe location in physical space 5000 over which focus point 5012 wouldhave been displayed.

FIGS. 6A-6C are flow diagrams illustrating method 600 of interactingwith an application for making measurements of a physical space using anaugmented reality environment in accordance with some embodiments.Method 600 is performed at an electronic device (e.g., portablemultifunction device 100 (FIG. 1A), device 300 (FIG. 3A), or computersystem 301 (FIG. 3B)) that includes a touch-sensitive display (e.g.,touch screen 112 (FIG. 1A), or display generation component(s) 304 incombination with input device(s) 302 (FIG. 3B)), and one or more cameras(e.g., optical sensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)),optionally one or more sensors to detect intensities of contacts withthe touch-sensitive display (e.g., contact intensity sensor(s) 165, FIG.1A), and optionally one or more tactile output generators (e.g., tactileoutput generator(s) 163 (FIG. 1A) or tactile output generator(s) 357(FIG. 3A)). Some operations in method 600 are, optionally, combinedand/or the order of some operations is, optionally, changed.

As described below, method 600 provides an intuitive way to reposition avirtual measurement point in an augmented reality-based measurement.Zooming in on an area that includes the measurement point in response toan input directed to the measurement point makes it easy to repositionthe measurement point more precisely. Method 600 reduces the number,extent, and/or nature of the inputs from a user when repositioning avirtual measurement point, thereby creating a more efficienthuman-machine interface. For battery-operated electronic devices,enabling a user to reposition measurement points faster and moreefficiently conserves power and increases the time between batterycharges.

The electronic device displays (602), on the touch-sensitive display, auser interface (e.g., user interface 5006, FIG. 5AX) of an application(e.g., an augmented reality measurement application or an applicationthat includes augmented reality measurement functionality).

The user interface includes (604) a representation of a field of view ofat least one of the one or more cameras (e.g., user interface 5006includes a live preview of the field of view of the camera of device100, FIG. 5AX). The representation of the field of view is displayed ata first magnification and updated over time based on changes to currentvisual data detected by at least one of the one or more cameras (e.g.,the representation of the field of view is a live view from at least oneof the one or more cameras). In addition, the field of view includes atleast a portion of a three-dimensional space (e.g., a space in thephysical world that includes physical objects). For example, the livepreview is displayed without zoom (or with a zoom factor of 1×) and isupdated as device 100 moves (e.g., as in FIGS. 5AU-5AV).

While displaying the representation of the field of view, the electronicdevice detects (606) a first touch input on the touch-sensitive display(e.g., a tap gesture or press input on an affordance which, whenactivated, adds a measurement point to the displayed representation ofthe field of view) (e.g., touch input 5158, FIG. 5AW).

In response to detecting the first touch input, the electronic deviceadds (608) and displays a (virtual) measurement point at a firstlocation in the representation of the field of view that corresponds toa first location in the three-dimensional space (e.g., measurement point5160, FIG. 5AW).

After adding the measurement point and while continuing to display therepresentation of the field of view (610), as at least one of the one ormore cameras moves, the electronic device displays (612) the measurementpoint at a location in the representation of the field of view thatcorresponds to the first location in the three-dimensional space. Forexample, as the position and/or orientation of at least one of the oneor more cameras change due to movement of the electronic device, the(virtual) measurement point continues to be displayed in the live viewat a location that corresponds to the first location in thethree-dimensional space, where the (virtual) measurement point wasinitially placed. In some embodiments, as at least one of the one ormore cameras moves, the displayed measurement point appears to beattached or anchored to the location in the three-dimensional spacewhere the (virtual) measurement point was initially placed.

After adding the measurement point and while continuing to display therepresentation of the field of view (610), the electronic device detects(614) a second touch input (e.g., touch input 5164, FIG. 5AY) at alocation on the touch-sensitive display that corresponds to a currentlocation of the measurement point in the representation of the field ofview (which, in turn, corresponds to the first location in thethree-dimensional space, where the measurement point was initiallyplaced).

In response to detecting the second touch input, the electronic deviceenlarges (616) display of at least a portion of the representation ofthe field of view from the first magnification to a secondmagnification, greater than the first magnification (e.g., the livepreview is enlarged by a zoom factor of 4× in FIG. 5AY). The enlargeddisplay of the portion of the representation of the field of viewincludes the measurement point. In some embodiments, in response to agesture on the measurement point (such as a tap, double tap, press,press and hold, or depinch gesture), the electronic device zooms in onan area that includes the measurement point, thereby enlarging an areathat includes the measurement point from a first magnification to asecond magnification. In some embodiments, zooming in on an area thatincludes the measurement point enables a user to reposition themeasurement point more precisely, e.g., in response to gestures directedto the zoomed-in measurement point.

In some embodiments, the one or more cameras are (618) located on theelectronic device adjacent to a portion of the touch-sensitive displaythat is near a side of the device on which the one or more cameras arepositioned (e.g., the one or more cameras are located in region 5008,FIG. 5A). In some embodiments, the one or more cameras are located onthe electronic device adjacent to a first portion of the touch-sensitivedisplay. In some embodiments, a surface of the touch-sensitive displayextends along an xy-plane, and the one or more cameras are adjacent tothe first portion of the touch-sensitive display along the xy-plane. Insome embodiments, the one or more cameras are adjacent to the firstportion of the touch-sensitive display in a z-direction that isperpendicular to the xy-plane. In some embodiments, the user interfaceincludes a first affordance that is displayed in a portion of thetouch-sensitive display (e.g., on a front side of the device) that ispositioned away from the one or more cameras on the back side of thedevice (e.g., away from a second fixed location on the touch-sensitivedisplay corresponding to a first fixed location on the back side of thedevice at which the one or more cameras are located), and which, whenactivated, adds a measurement point to the displayed representation ofthe field of view (e.g., measurement addition button 5014, FIG. 5A). Insome embodiments, the first affordance is displayed in a second portionof the touch-sensitive display, where the second portion of thetouch-sensitive display is distinct from the first portion of thetouch-sensitive display, and where the second portion of thetouch-sensitive display is located away from the side of the device onwhich the one or more cameras are positioned. In some embodiments, theuser interface further includes one or more second affordances (e.g.,buttons 5018, 5020, 5022, FIG. 5A) displayed in accordance with a firstorientation of the electronic device (e.g., portrait orientation, asshown in FIG. 5A). In some embodiments, the electronic device detectsmovement (e.g., rotation) of the electronic device to a secondorientation (e.g., rotation to a landscape orientation as shown in FIG.5B or FIG. 5C). In some embodiments, in response to detecting themovement of the electronic device to the second orientation, theelectronic device updates display of the one or more second affordancesin accordance with the second orientation of the electronic device(e.g., buttons 5018, 5020, and 5022 move to different regions of theuser interface in FIG. 5B or FIG. 5C without regard to the positions ofthe one or more cameras) and continues to display the first affordancein the portion of the touch-sensitive display that is away from the sideof the device on which the one or more cameras are positioned (e.g.,measurement addition button 5014 is displayed away from region 5008 inFIGS. 5B and 5C).

In some embodiments, in the second orientation, the electronic deviceupdates display of the first affordance such that the first affordanceis displayed at a different position within the portion of thetouch-sensitive display that is away from the side of the device onwhich the one or more cameras are positioned (e.g., at a position withina predefined distance of an edge or a corner of the touch-sensitivedisplay, to facilitate activation of the first affordance duringsingle-handed operation of the electronic device) (e.g., measurementaddition button 5014). In some embodiments, the first affordance isrestricted to positions within the portion of the touch-sensitivedisplay that is away from the side of the device on which the one ormore cameras are positioned, so as to deter a user from holding theelectronic device in a manner in which the user's hand obscures thefield of view. In some embodiments, while the electronic device is in afirst orientation (e.g., a first landscape orientation) in which the oneor more cameras are located on a left half of the electronic device, thefirst affordance is displayed in the user interface on a right half ofthe electronic device; and, while the electronic device is in a secondorientation (e.g., a second landscape orientation in which the device isrotated 180 degrees from the first landscape orientation) in which theone or more cameras are located on the right half of the electronicdevice, the first affordance is displayed in the user interface on theleft half of the electronic device (e.g., as shown in and describedherein with reference to FIGS. 5B-5C).

When the device orientation changes, automatically keeping an affordancethat is used to place measurement points at a location on that touchsensitive display that is away from the one or more cameras reduces thechance that a user will hold the electronic device in a way thatobscures the field of view of a camera that is providing the live view.Automatically repositioning an affordance in this manner when the deviceorientation changes enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device, such as obscuring the field ofview).

In some embodiments, the user interface of the application includes(620) one or more affordances that correspond to operations of theapplication, including a capture affordance (e.g., an affordance such asa virtual shutter button which, when activated, initiates capture ofmedia that corresponds to the representation of the field of view)(e.g., media capture button 5016, FIG. 5A). In some embodiments, theelectronic device detects a third touch input on the capture affordance,and, in accordance with a determination that the third touch input meetsfirst media capture criteria, initiates capture of media of a first type(e.g., a video or a live photo) that includes a sequence of images ofthe field of view of at least one of the one or more cameras (and, insome embodiments, corresponding audio) (e.g., as described herein withreference to FIGS. 5BJ-5BK). In some embodiments, the first mediacapture criteria include a first requirement that the third touch inputbe maintained on the touch-sensitive display for at least a predefinedthreshold amount of time, and a second requirement that an amount ofmovement of the third touch input across the touch-sensitive display beless than a predefined threshold amount of movement. In someembodiments, the first media capture criteria are satisfied by astationary long-press or a press-and-hold gesture on the captureaffordance.

In some embodiments, the captured sequence of images includes (virtual)measurement information displayed over the images (e.g., one or more(virtual) measurement points, lines between the measurement points,labels for the measurement points, and/or distances between measurementpoints) (e.g., as described herein with reference to FIGS. 5BJ-5BK). Insome embodiments, the captured sequence of images does not includedisplay of other affordances (besides the measurement informationdisplayed over the images) that are shown in the user interface of theapplication, such as the capture affordance. In other words, in someembodiments, instead of capturing a screen recording of everything beingshown in the user interface of the application, the device just capturesa video of the field of view with the (virtual) measurement informationsuperimposed upon the field of view (e.g., as described herein withreference to FIGS. 5BJ-5BK).

Providing a virtual shutter button or other capture affordance makes iteasy to record a video of the objects being measured, along with thevirtual measurement information that is displayed over the objects.Recording such a video, without also recording other elements in theuser interface of the application, enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to create recordings that show the objects and themeasurements, without showing extraneous information that was displayedin the user interface during the recording).

In some embodiments, in accordance with a determination that the thirdtouch input meets second media capture criteria, the electronic deviceinitiates (622) capture of media of a second type (e.g., a still image)that includes a single image of the field of view of at least one of theone or more cameras (e.g., without corresponding audio) (e.g., as shownin and described herein with reference to FIGS. 5BH-5BI). In someembodiments, the second media capture criteria include a firstrequirement that the third touch input cease to be detected on thetouch-sensitive display before a predefined threshold amount of timeelapses, and a second requirement that an amount of movement of thesecond touch input across the touch-sensitive display be less than apredefined threshold amount of movement. In some embodiments, the secondmedia capture criteria are satisfied by a stationary tap gesture on thecapture affordance.

In some embodiments, the captured still image does not include displayof other affordances (besides the measurement information displayed overthe images) that are shown in the user interface of the application,such as the capture affordance (e.g., as shown in and described hereinwith reference to FIGS. 5BH-5BI). In other words, in some embodiments,instead of capturing a still image of everything being shown in the userinterface of the application, the device just captures a still image ofthe field of view with the (virtual) measurement informationsuperimposed upon the field of view.

Providing a capture affordance that can initiate capture of either astill image or a video makes it easy to obtain either a still image or avideo of the objects being measured, along with the virtual measurementinformation that is displayed over the objects. Obtaining such a stillimage or video, without also including other elements in the userinterface of the application, enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to create a still image or video that shows the objects and themeasurements, without showing extraneous information that was displayedin the user interface during the recording). In addition, providingadditional control options for the capture affordance (e.g., to captureeither a still image or a video, depending on the characteristics of thetouch input on the capture affordance), without cluttering the userinterface with additional displayed controls, enhances the operabilityof the device and makes the user-device interface more efficient.

In some embodiments, prior to displaying the user interface of theapplication, the electronic device displays (624) a control panel userinterface that includes a plurality of controls (e.g., control centeruser interface 5188, FIG. 5BN), where a first control in the pluralityof controls corresponds to the application (e.g., augmented realitymeasurement application icon 5190, FIG. 5BN). In some embodiments, theelectronic device detects a touch input (e.g., a tap gesture or pressinput) activating the first control, and, in response to detecting thetouch input activating the first control, displays the user interface ofthe application (e.g., as described herein with reference to FIG. 5BN).Providing access to a measurement application via a control panel makesit easier to find and launch the application. Reducing the number ofinputs needed to find and launch an application enhances the operabilityof the device and makes the user-device interface more efficient.

In some embodiments, the user interface is (626) a first user interfaceof a plurality of user interfaces in the application. In someembodiments, the first user interface corresponds to a measurement modeof the application. In some embodiments, a second user interface of theplurality of user interfaces corresponds to a levelling mode of theapplication. In some embodiments, an augmented reality measurementapplication or an application that includes augmented realitymeasurement functionality also includes level functionality. Providingboth measurement and level functionality in the same application makesit easier to find and use these related functions. Providing multiplerelated functionalities in the same application enhances the operabilityof the device and makes the user-device interface more efficient.

In some embodiments, the electronic device determines (628) a distancebetween the electronic device and the first location in thethree-dimensional space that corresponds to the measurement point (e.g.,a distance from one of the cameras of the electronic device to the firstlocation in the three-dimensional space that corresponds to themeasurement point). In some embodiments, the distance from one of thecameras to the first location in the three-dimensional space isdetermined based on depth information captured by at least one of theone or more cameras (e.g., by depth sensors that are optionally part ofthe one or more cameras) and/or based on disparity information betweenmultiple different cameras (e.g., determined by comparing informationcaptured by multiple different cameras). In some embodiments, inaccordance with a determination that the determined distance is lessthan a first threshold distance, a magnification factor between thefirst magnification and the second magnification (e.g., an amount ofincrease in magnification from the first magnification to the secondmagnification) is a first magnification factor (e.g., corresponding to aminimum amount of zoom, that does not change as the distance between theelectronic device and the first location in the three-dimensional spacedecreases below the first threshold distance). In some embodiments, inaccordance with a determination that the determined distance is greaterthan a second threshold distance, the magnification factor is a secondmagnification factor (e.g., corresponding to a maximum amount of zoom,that does not change as the distance between the electronic device andthe first location in the three-dimensional space increases above thesecond threshold distance). In some embodiments, in accordance with adetermination that the determined distance is between the firstthreshold distance and the second threshold distance, the magnificationfactor is a respective magnification factor, between the firstmagnification factor and the second magnification factor, that dependson the determined distance. For example, as shown in and describedherein with reference to FIGS. 5AY-5BE, an amount of zoom of the livepreview is based on the distance between device 100 and table 5002,optionally with a maximum and/or a minimum limit on the amount of zoom.

More generally, in some embodiments, the amount of zoom is increasedwhen the electronic device is further from the point of interest in thethree-dimensional space (optionally subject to a maximum amount ofzoom), and the amount of zoom is decreased when the electronic device iscloser to the point of interest in the three-dimensional space(optionally subject to a minimum amount of zoom). In principle, agreater amount of zoom is needed when the electronic device is furtherfrom the point(s) of interest in the three-dimensional space, becausefeatures of interest are more difficult to discern (e.g., in therepresentation of the field of view) at greater distances, whereas alesser amount of zoom is needed when the electronic device is closer tothe point(s) of interest in the three-dimensional space, becausefeatures of interest are more readily perceived (e.g., in therepresentation of the field of view) at lesser distances. In someembodiments, a maximum amount of zoom is imposed in accordance withhardware specifications (e.g., zoom limitations) of the one or morecameras.

Automatically varying the amount of zooming based on the distance fromthe electronic device to a location in the three-dimensional space thatcorresponds to the measurement point makes it easier to reposition themeasurement point, because the measurement point is displayed at anappropriate level of magnification for each distance. Performing a zoomoperation with different magnification factors, which depend on anautomatically measured distance, without requiring further user input,enhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by automatically displaying themeasurement point at a magnification level where proper inputs can beprovided to reposition the measurement point).

In some embodiments, while displaying the enlarged display of at leastthe portion of the representation of the field of view, the electronicdevice detects (630) a fourth touch input that includes detecting acontact at the current location of the measurement point and detectingmovement of the contact across the touch-sensitive display (e.g., touchinput 5164, FIG. 5AZ). In some embodiments, the second touch input andthe fourth touch input are different portions of an input by a singlecontinuous contact (e.g., the second touch input is a first portion ofthe input that touches down the contact on the touch-sensitive display,and the fourth input is a second portion of the input that includesmovement of the contact). In some embodiments, in response to detectingthe movement of the contact across the touch-sensitive display, theelectronic device moves the measurement point across the representationof the field of view in accordance with the movement of the contact inthe fourth touch input (e.g., as described herein with reference tomeasurement point 5160, FIG. 5AZ). Dragging the measurement point whileviewing an enlarged area around the measurement point makes it easy toreposition the measurement point precisely with movement of a singlecontact. Reducing the number of inputs needed to perform a repositioningoperation enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, the first touch input is (632) detected on anaffordance which, when activated, adds a measurement point to thedisplayed representation of the field of view. In some embodiments, theaffordance is displayed in the user interface at a (fixed) predefinedlocation. In some embodiments, a location in the representation of thefield of view where the measurement point is initially added, inresponse to activation of the affordance, is distinct from thepredefined location of the affordance. For example, in FIGS. 5AV-5AW,measurement point 5160 is added to user interface 5006 at a locationthat is away from the location of measurement addition button 5014.Providing an affordance for adding measurement points that is locatedaway from where the added measurement points are initially displayedmakes it easy to see where a measurement point will be placed.Displaying an affordance for adding points away from the location wherethe points are initially added enhances the operability of the deviceand makes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs for adding measurement points and reducinguser mistakes when operating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 6A-6C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 800, 900, 1000, 1100, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 600 described above withrespect to FIGS. 6A-6C. For example, the inputs, user interface elements(e.g., measurement points, measurement segments, virtual annotations,representations of the physical space or field of view, affordances,alerts, indicators, labels, anchor points, and/or placement userinterface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 600optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods700, 800, 900, 1000, 1100, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 7A-7E are flow diagrams illustrating method 700 of addingmeasurements to a displayed representation of a physical space in anaugmented reality environment in accordance with some embodiments.Method 700 is performed at an electronic device (e.g., portablemultifunction device 100 (FIG. 1A), device 300 (FIG. 3A), or computersystem 301 (FIG. 3B)) that includes a touch-sensitive display (e.g.,touch screen 112 (FIG. 1A), or display generation component(s) 304 incombination with input device(s) 302 (FIG. 3B)), one or more sensors todetect intensities of contacts with the touch-sensitive display (e.g.,contact intensity sensor(s) 165, FIG. 1A), and one or more cameras(e.g., optical sensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)), andoptionally one or more tactile output generators (e.g., tactile outputgenerator(s) 163 (FIG. 1A) or tactile output generator(s) 357 (FIG.3A)). Some operations in method 700 are, optionally, combined and/or theorder of some operations is, optionally, changed.

As described below, method 700 provides an intuitive way to moreprecisely add virtual measurement points and segments in augmentedreality-based measurements. For an electronic device with atouch-sensitive display, one or more sensors to detect intensities ofcontacts with the touch-sensitive display, and one or more cameras,repeated presses by a continuously detected contact on thetouch-sensitive display make it easier to hold the electronic devicesteady (e.g., as compared to repeated taps on the touch-sensitivedisplay) while positioning the locations of the measurement points witha live view from at least one of the cameras. Method 700 changes thenature of the inputs from a user when adding virtual measurement pointsand segments, thereby creating a more efficient human-machine interface.For battery-operated electronic devices, enabling a user to addmeasurement points and segments more accurately and more efficientlyconserves power and increases the time between battery charges.

The electronic device displays (702), on the touch-sensitive display, auser interface of an application (e.g., an augmented reality measurementapplication or an application that includes augmented realitymeasurement functionality).

The user interface includes (704) a representation of a field of view ofat least one of the one or more cameras. The representation of the fieldof view is updated over time based on changes to current visual datadetected by at least one of the one or more cameras (e.g., therepresentation of the field of view is a live view, which changes as theone or more cameras move and/or as the physical world in the field ofview of the one or more cameras change). The user interface includes ameasurement-point-creation indicator that is displayed over therepresentation of the field of view (e.g., reticle 5010 in conjunctionwith focus point 5012, FIG. 5AH which are displayed within userinterface 5006 on top of the live preview of the camera). In addition,the field of view includes at least a portion of a three-dimensionalspace (e.g., a space in the physical world that includes physicalobjects) (e.g., physical space 5000).

The electronic device detects (706) a contact on the touch-sensitivedisplay (e.g., touch input 5120, FIG. 5AI).

While continuously detecting the contact on the touch-sensitive display(708), and while the measurement-point-creation indicator is displayedover a first location in the representation of the field of view thatcorresponds to a first location in the three-dimensional space (710), inaccordance with a determination that first criteria (e.g.,measurement-point-creation criteria) are met, where the first criteriainclude a requirement that an intensity of the contact meet (e.g., reachor exceed) a respective intensity threshold in order for the firstcriteria to be met, the electronic device adds (712) and displays afirst (virtual) measurement point (e.g., measurement point 5122, FIG.5AI) in the representation of the field of view that corresponds to thefirst location in the three-dimensional space. In some embodiments, theelectronic device determines whether the first criteria are met. In someembodiments, the first criteria are met when the intensity of thecontact exceeds an intensity threshold (e.g., a light press intensitythreshold IT_(L), which is above a contact detection intensity thresholdIT₀) (e.g., as indicated by intensity meter 5040, FIG. 5AI). In someembodiments, the first criteria are met when the intensity of thecontact falls below the intensity threshold (after exceeding theintensity threshold). In some embodiments, the first criteria include arequirement that, when other criteria of the first criteria are met, thecontact be positioned on an affordance, which, when activated, adds ameasurement point to the displayed representation of the field of viewat the location over which the measurement-point-creation indicator isdisplayed (e.g., measurement addition button 5014, FIG. 5AI).

While continuously detecting the contact on the touch-sensitive display(708), and after adding the first (virtual) measurement point, theelectronic device updates (714) the representation of the field of viewas the electronic device is moved (e.g., as the field of view of the oneor more cameras changes). In some embodiments, in response to theelectronic device being moved, the electronic device displays a dynamicmeasurement segment between the first (virtual) measurement point in therepresentation of the field of view (e.g., while the field of view ofthe one or more cameras includes the first location in thethree-dimensional space) and the measurement-point-creation indicator inthe user interface (e.g., dynamic measurement segment 5128, FIG. 5AK).In some embodiments, the display of the dynamic measurement segmentchanges in accordance with the movement of the device (e.g., whichchanges the field of view of the one or more cameras). For example, whenthe measurement-point-creation indicator is centered at a fixed positionwithin the representation of the field of view, a distance between thefirst measurement point in the representation of the field of view and alocation in the representation of the field of view that corresponds tothe measurement-point-creation indicator changes as the device (and thefield of view of the one or more cameras) moves relative to the firstlocation in the three-dimensional space and, accordingly, a length ofthe dynamic measurement segment changes as the device moves.

While continuously detecting the contact on the touch-sensitive display(708), after the electronic device is moved, and while themeasurement-point-creation indicator is displayed over a second locationin the representation of the field of view that corresponds to a secondlocation in the three-dimensional space (716), in accordance with adetermination that the first criteria are met while themeasurement-point-creation indicator is displayed over the secondlocation in the representation of the field of view that corresponds tothe second location in the three-dimensional space (718): the electronicdevice adds (720) and displays a second (virtual) measurement point(e.g., as explained herein with respect to the addition of measurementpoint 5132, FIG. 5AL) in the representation of the field of view thatcorresponds to the second location in the three-dimensional space. Insome embodiments, the second location in the representation of the fieldof view is the same as the first location in the representation of thefield of view, for example when the measurement-point-creation indicatoris centered at a fixed position within the representation of the fieldof view. In some embodiments, the electronic device determines whetherthe first criteria are met. In addition, the electronic device displays(722) a first measurement segment connecting the first measurement pointand the second measurement point.

In some embodiments, the first measurement segment (connecting the firstmeasurement point and the second measurement point) is displayed inaccordance with a determination that second criteria (e.g.,measurement-segment-creation criteria) are met. In some embodiments, thesecond criteria include a requirement that, following a respectiveinstance when the first criteria are met for adding and displaying arespective measurement point, the contact be maintained on thetouch-sensitive display (e.g., on an affordance which, when activated,adds a measurement point to the displayed representation of the field ofview at the location over which the measurement-point-creation indicatoris displayed) until a next instance that the first criteria are met foradding and displaying a next measurement point (e.g., as shown in anddescribed herein with reference to FIGS. 5AH-5AS). That is, the secondcriteria include a requirement that the contact be maintained betweenthe creation of successive measurement points that satisfy the(intensity-based) first criteria. In some embodiments, the intensitythreshold for the first criteria is a second intensity threshold (e.g.,a light press intensity threshold IT_(L)) that is above a firstintensity threshold (e.g., a contact detection intensity threshold(IT₀), and the second criteria include a requirement that an intensityof the contact remain at or above the first intensity threshold whilethe contact is maintained. In some embodiments, the second criteriainclude a requirement that an intensity of the contact decrease to orbelow the second intensity threshold while the contact is maintained(e.g., if the first criteria require that the intensity of the contactexceed the second intensity threshold). In some embodiments, the secondcriteria include a requirement that an intensity of the contact remainat or below the second intensity threshold while the contact ismaintained (e.g., if the first criteria require that the intensity ofthe contact fall below the intensity threshold (after exceeding theintensity threshold)).

More generally, in some embodiments, after a respective point is addedand displayed in accordance with the first criteria being met, as longas the contact is maintained on the touch-sensitive display, one or moreadditional measurement points, as well as their correspondingmeasurement segments, are added and displayed in accordance with thefirst criteria being met for each additional measurement point. That is,as long as the contact is maintained on the touch-sensitive-display,each subsequent instance when the first criteria is met adds both anadditional point and an additional measurement segment between the(newly-added) additional point and the most-recently-added prior point.Stated another way, after a respective point is added in accordance withthe first criteria being met, the electronic device operates in acontinuous measurement-(point-and-segment-)creation mode until thecontact ceases to be maintained on the touch-sensitive display.

In some embodiments, while continuously detecting the contact on thetouch-sensitive display, and after adding the second (virtual)measurement point, the electronic device updates (724) therepresentation of the field of view as the electronic device is movedagain (e.g., as shown in and described herein with reference to FIGS.5AN-5AQ). In some embodiments, while continuously detecting the contacton the touch-sensitive display, after the electronic device is movedagain, and while the measurement-point-creation indicator is displayedover a third location in the representation of the field of view thatcorresponds to a third location in the three-dimensional space, inaccordance with a determination that the first criteria are met whilethe measurement-point-creation indicator is displayed over the thirdlocation in the representation of the field of view that corresponds tothe third location in the three-dimensional space, the electronic deviceadds and displays a third (virtual) measurement point in therepresentation of the field of view that corresponds to the thirdlocation in the three-dimensional space (e.g., measurement point 5142,FIG. 5AR), and displays a second measurement segment connecting thesecond measurement point and the third measurement point (e.g.,completed measurement segment 5136, FIG. 5AR). In some embodiments, thethird location in the representation of the field of view is the same asthe first location in the representation of the field of view, forexample when the measurement-point-creation indicator is centered at afixed position within the representation of the field of view.

In some embodiments, in accordance with a determination that the firstcriteria are not met while the measurement-point-creation indicator isdisplayed over the second location in the representation of the field ofview that corresponds to the second location in the three-dimensionalspace, the electronic device forgoes (726) adding and displaying thesecond (virtual) measurement point in the representation of the field ofview that corresponds to the second location in the three-dimensionalspace, and forgoes displaying the first measurement segment connectingthe first measurement point and the second measurement point.

In some embodiments, after adding the second (virtual) measurementpoint, the electronic device updates (728) the representation of thefield of view as the electronic device is moved again. In someembodiments, after the electronic device is moved again, while themeasurement-point-creation indicator is displayed over a third locationin the representation of the field of view that corresponds to the thirdlocation in the three-dimensional space, and in accordance with adetermination that the first criteria are met while themeasurement-point-creation indicator is displayed over the thirdlocation in the representation of the field of view that corresponds tothe third location in the three-dimensional space, the electronic deviceadds and displays a third (virtual) measurement point in therepresentation of the field of view that corresponds to the thirdlocation in the three-dimensional space. In some embodiments, inaccordance with a determination that second criteria (e.g.,measurement-segment-creation criteria) are met, where the secondcriteria include a requirement that the contact is maintained betweenthe first criteria being met for adding the second measurement point andthe first criteria being met for adding the third measurement point, theelectronic device displays a second measurement segment connecting thesecond measurement point and the third measurement point. In someembodiments, in accordance with a determination that the second criteriaare not met, the electronic device forgoes displaying the secondmeasurement segment connecting the second measurement point and thethird measurement point (e.g., as shown in and described herein withreference to FIGS. 5AH-5AS). In some embodiments, if a respectivemeasurement point is a most-recently-added measurement point (e.g., thesecond measurement point) and is an endpoint of a measurement segmentthat is the most-recently-added measurement segment (e.g., the firstmeasurement segment), then, in accordance with a determination that thefirst criteria (e.g., the measurement-point-creation criteria) are metand the second criteria are not met, the electronic device adds anddisplays an additional measurement point (e.g., the third measurementpoint) and forgoes displaying an additional measurement segment betweenthe respective measurement point (the most-recently-added measurementpoint prior to displaying the additional measurement point) and theadditional measurement point.

In some embodiments, while continuously detecting the contact on thetouch-sensitive display, and while the electronic device is being moved,a dynamic measurement segment is displayed between a most-recently-addedmeasurement point (e.g., the second measurement point) and a location inthe representation of the field of view that corresponds to themeasurement-point-creation indicator (e.g., a dynamic measurementsegment as described herein with reference to operation 714), and thedynamic measurement segment continues to be displayed in accordance witha determination that the second criteria are met. In some embodiments,if the most-recently-added measurement point is an endpoint of anothermeasurement segment that is the most-recently-added measurement segmentprior to displaying the dynamic measurement segment (e.g., the firstmeasurement segment), then the electronic device ceases to display thedynamic measurement segment in response to liftoff of the contact afterthe first criteria are met for adding the second measurement point(e.g., at any point before the first criteria are met for adding thethird measurement point) (e.g., as shown in and described herein withreference to FIGS. 5AH-5AS).

Adding a measurement point with or without a corresponding measurementsegment to the immediately prior measurement point, depending on whethera contact has been maintained on the touch sensitive display, providesan additional option for adding a measurement point without adding ameasurement segment (e.g., a user lifts off the contact to indicate thatthe next measurement point to be added is not connected to the priormeasurement points and segments). Providing additional control optionswithout cluttering the UI with additional displayed controls enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, the electronic device includes one or more tactileoutput generators. In some embodiments, in accordance with adetermination that the intensity of the contact meets (e.g., reaches(increases to) or exceeds) the respective intensity threshold, theelectronic device generates (730) a tactile output (e.g., to indicatethat the intensity of the contact is sufficient to add a measurementpoint) (e.g., as described herein with reference to tactile output 5124,FIG. 5AI). In some embodiments, a tactile output is generated inaccordance with the determination that the first criteria are met (e.g.,that other requirements of the first criteria are also met, in additionto the requirement that the intensity of the contact meet the respectiveintensity threshold). In some embodiments, the electronic deviceprovides a tactile output when the respective intensity threshold is metto indicate that the intensity of the contact is sufficient to add ameasurement point and/or to indicate that the measurement point has beenadded. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments in which the electronic device includes one or moretactile output generators, the respective intensity threshold is (732) asecond intensity threshold (e.g., a light press intensity thresholdIT_(L)) that is above a first intensity threshold (e.g., a contactdetection intensity threshold IT₀). In some embodiments, in accordancewith a determination that the intensity of the contact ceases to meetthe second intensity threshold (after meeting the second intensitythreshold) (e.g., the intensity of the contact decreases to or below thesecond intensity threshold), and that the intensity of the contact meetsthe first intensity threshold (e.g., the intensity of the contactremains at or above the first intensity threshold), the electronicdevice generates a tactile output (e.g., to indicate addition of ameasurement point) (e.g., as described herein with reference to FIG.5AJ). In some embodiments, the electronic device provides a tactileoutput when the intensity falls below the second intensity threshold toindicate that a measurement point has been added. Providing improvedfeedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, the adding and the displaying of the firstmeasurement point is (734) performed when the intensity of the contactmeets the respective intensity threshold, while themeasurement-point-creation indicator is displayed over the firstlocation in the representation of the field of view that corresponds tothe first location in the three-dimensional space (e.g., as shown in anddescribed herein with reference to FIG. 5AI). In some embodiments, theadding and the displaying of the second measurement point is performedwhen the intensity of the contact meets the respective intensitythreshold after the electronic device is moved, while themeasurement-point-creation indicator is displayed over the secondlocation in the representation of the field of view that corresponds tothe second location in the three-dimensional space.

In some embodiments, the respective intensity threshold is (736) asecond intensity threshold (e.g., a light press intensity thresholdIT_(L)) that is above a first intensity threshold (e.g., a contactdetection intensity threshold IT₀). In some embodiments, the firstcriteria include a requirement that the intensity of the contact fallsbelow the respective intensity threshold, after meeting the respectiveintensity threshold, in order for the first criteria to be met. In someembodiments, the adding and the displaying of the first measurementpoint is performed when the intensity of the contact falls below therespective intensity threshold, while the measurement-point-creationindicator is displayed over the first location in the representation ofthe field of view that corresponds to the first location in thethree-dimensional space. In some embodiments, the adding and thedisplaying of the second measurement point is performed when, after theelectronic device is moved and while the measurement-point-creationindicator is displayed over the second location in the representation ofthe field of view that corresponds to the second location in thethree-dimensional space, the intensity of the contact falls below therespective intensity threshold.

In some embodiments, while displaying the representation of the field ofview of the one or more cameras, the electronic device determines (738)an anchor point in the representation of the field of view of the one ormore cameras that corresponds to a respective location in thethree-dimensional space. In some embodiments, as the one or more camerasmove, while the measurement-point-creation indicator (or at least aportion thereof) is over (or proximate to) the anchor point, theelectronic device changes a visual appearance of themeasurement-point-creation indicator to indicate that a respectivemeasurement point will be added at the anchor point if the contact meetsthe first criteria (e.g., as described herein with reference to reticle5010 and focus point 5012 in FIG. 5AH). Providing visual feedback that ameasurement point will be added at the anchor point if the contact meetsthe first criteria makes it easy to add a measurement point at theanchor point. Providing improved feedback enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the user interface includes (740) an affordance,which, when activated, adds a (virtual) measurement point in therepresentation of the field of view at a location in the representationof the field of view over which the measurement-point-creation indicatoris displayed (e.g., measurement addition button 5014). In someembodiments, the electronic device detects a touch input (e.g., a tapgesture) activating the affordance, and, in response to detecting thetouch input activating the affordance, adds and displays a measurementpoint in the representation of the field of view at the location in therepresentation of the field of view over which themeasurement-point-creation indicator is displayed (e.g., as shown in anddescribed herein with reference to FIG. 5K).

It should be understood that the particular order in which theoperations in FIGS. 7A-7E have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 800, 900, 1000, 1100, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 700 described above withrespect to FIGS. 7A-7E. For example, the inputs, user interface elements(e.g., measurement points, measurement segments, virtual annotations,representations of the physical space or field of view, affordances,alerts, indicators, labels, anchor points, and/or placement userinterface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 700optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 800, 900, 1000, 1100, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 8A-8C are flow diagrams illustrating method 800 of adding virtualmeasurement points at automatically determined anchor points in anaugmented reality environment in accordance with some embodiments.Method 800 is performed at an electronic device (e.g., portablemultifunction device 100 (FIG. 1A), device 300 (FIG. 3A), or computersystem 301 (FIG. 3B)) that includes a touch-sensitive display (e.g.,touch screen 112 (FIG. 1A), or display generation component(s) 304 incombination with input device(s) 302 (FIG. 3B)), and one or more cameras(e.g., optical sensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)),optionally one or more sensors to detect intensities of contacts withthe touch-sensitive display (e.g., contact intensity sensor(s) 165, FIG.1A), and optionally one or more tactile output generators (e.g., tactileoutput generator(s) 163 (FIG. 1A) or tactile output generator(s) 357(FIG. 3A)). Some operations in method 800 are, optionally, combinedand/or the order of some operations is, optionally, changed.

As described below, method 800 provides an intuitive way to add virtualmeasurement points in augmented reality-based measurements, either atautomatically determined anchor points or away from such anchor points.An electronic device provides visual feedback that a measurement pointwill be added at an anchor point if measurement-point-creation criteria,which makes it easy to add a measurement point at the anchor point.Providing improved feedback enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device). For battery-operated electronicdevices, enabling a user to add measurement points at automaticallydetermined anchor points (or add measurement points away from suchanchor points) faster and more efficiently conserves power and increasesthe time between battery charges.

The electronic device displays (802), on the touch-sensitive display, auser interface of an application (e.g., an augmented reality measurementapplication or an application that includes augmented realitymeasurement functionality).

The user interface includes (804) a representation of a field of view ofat least one of the one or more cameras. The representation of the fieldof view is updated over time based on changes to current visual datadetected by at least one of the one or more cameras (e.g., therepresentation of the field of view is a live view). In addition, theuser interface includes a measurement-point-creation indicator that isdisplayed over the representation of the field of view (e.g., reticle5010 in combination with focus point 5012, FIG. 5K). The field of viewincludes at least a portion of a three-dimensional space (e.g., a spacein the physical world that includes physical objects).

While displaying the representation of the field of view (806), theelectronic device determines (808) an anchor point at a location in therepresentation of the field of view that corresponds to a first locationin the three-dimensional space. In some embodiments, the electronicdevice determines a plurality of anchor points that correspond to aplurality of locations in the three-dimensional space, such as a cornerof a physical object in the three-dimensional space, points along anedge of a physical object in the three-dimensional space, or the like.

While displaying the representation of the field of view (806), as atleast one of the one or more cameras move, and while themeasurement-point-creation indicator (or at least a portion thereof) isover (or proximate to) the anchor point, the electronic device changes(810) a visual appearance of the measurement-point-creation indicator toindicate that a (virtual) measurement point will be added at the anchorpoint if a touch input meets first criteria (e.g., measurement pointcreation criteria) (e.g., as described herein with reference to reticle5010 and focus point 5012 in FIG. 5AK).

While displaying the representation of the field of view (806), theelectronic device detects (812) a first touch input on thetouch-sensitive display that meets the first criteria (e.g., a tapgesture on an affordance or a hard press input on an affordance, thehard press input meeting or exceeding an intensity threshold) (e.g.,touch input 5038, FIG. 5K.

While displaying the representation of the field of view (806), inresponse to detecting the first touch input that meets the firstcriteria (814), and in accordance with a determination that themeasurement-point-creation indicator (or at least a portion thereof) isover (or proximate to) the anchor point when the first criteria are met(816), the electronic device adds and displays a first (virtual)measurement point at the anchor point in the representation of the fieldof view that corresponds to the first location in the three-dimensionalspace (e.g., measurement point 5042, FIG. 5K).

While displaying the representation of the field of view (806), inaccordance with a determination that the measurement-point-creationindicator (or at least a portion thereof) is not over (or proximate to)the anchor point when the first criteria are met (818), the electronicdevice adds and displays a first (virtual) measurement point at a firstlocation in the representation of the field of view that is away fromthe anchor point (e.g., at a location in the representation of the fieldof view that does not correspond to the first location in thethree-dimensional space).

In some embodiments, the determined anchor point is (820) also anendpoint of a (currently) displayed representation of a measurement(e.g., as described herein with respect to the anchor pointcorresponding to measurement point 5054, FIG. 5Q), and adding ameasurement point at the anchor point, if a touch input meets firstcriteria, will not form a region, in the representation of the field ofview, that is enclosed by a plurality of displayed measurement segments(e.g., and their associated endpoints). For example, a measurementsegment that will be added in conjunction with or in response to addingthe measurement point at the determined anchor point will not form aclosed polygon that includes the added measurement segment as a finalside of the closed polygon). Providing visual feedback that ameasurement point will be added at the anchor point ifmeasurement-point-creation criteria are met makes it easy to add ameasurement point at the anchor point, including at an anchor point thatis not just closing a loop of measurement points. Providing improvedfeedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, the electronic device displays (822), over therepresentation of the field of view, a representation of a firstmeasurement. The representation of the first measurement includes afirst endpoint that corresponds to a second location in thethree-dimensional space, a second endpoint that corresponds to a thirdlocation in the three-dimensional space, and a first line segmentconnecting the first endpoint and the second endpoint. In addition, thedetermined anchor point is a midpoint of the first line segment (e.g.,the first location, to which the anchor point corresponds, is halfwaybetween the second location and the third location in thethree-dimensional space) (e.g., as shown in and described herein withreference to FIG. 5AA). Providing visual feedback that a measurementpoint will be added at an anchor point that is at the midpoint of ameasurement segment if measurement-point-creation criteria are met makesit easy to add a measurement point at the midpoint of a measurementsegment. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the electronic device includes one or more tactileoutput generators. In some embodiments, as at least one of the one ormore cameras moves, and while the measurement-point-creation indicator(or at least a portion thereof) is over (or proximate to) the anchorpoint, the electronic device generates (824) a tactile output inconjunction with changing the visual appearance of themeasurement-point-creation indicator (e.g., as described herein withrespect to FIG. 5H). In some embodiments, adding a measurement point atthe anchor point, if a touch input meets first criteria, will not form aregion enclosed by a plurality of displayed measurement segments (e.g.,and their associated endpoints) in the representation of the field ofview of the one or more cameras. For example, a measurement segment thatwill be added in conjunction with or in response to adding themeasurement point will not form a closed polygon that includes the addedmeasurement segment as a final side of the closed polygon. In someembodiments, a tactile output is generated when snapping to anidentified physical feature in the three-dimensional space. In someembodiments, a tactile output is generated when snapping to a currentlydisplayed representation of a measurement, to a respective measurementpoint (e.g., an endpoint) thereof, and/or to a midpoint of a measurementsegment thereof. Providing both haptic and visual feedback that ameasurement point will be added at the anchor point ifmeasurement-point-creation criteria are met makes it easy to add ameasurement point at the anchor point, including at an anchor point thatis not just closing a loop of measurement points. Providing improvedfeedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments in which the electronic device includes one or moretactile output generators, the electronic device detects (826) movementof the measurement-point-creation indicator away from the anchor point(e.g., so that the measurement-point-creation indicator is not over (ornot proximate to) the anchor point) (e.g., due to movement of the one ormore cameras that changes the field of view while themeasurement-point-creation indicator remains at a fixed location withinthe user interface). In some embodiments, in response to detecting themovement of the measurement-point-creation indicator away from theanchor point, the electronic device generates a tactile output (e.g., asdescribed herein with respect to FIG. 5I). Providing haptic feedbackthat a measurement point will not be added at the anchor point ifmeasurement-point-creation criteria are met helps guide a user whileplacing measurement points. Providing improved feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments in which the electronic device includes one or moretactile output generators, in response to detecting the first touchinput that meets the first criteria, the electronic device adds (828)the first measurement point without adding a measurement segmentconnected to the first measurement point (e.g., the first measurementpoint is a first endpoint of a new measurement) and generates a firsttactile output (e.g., a start-of-measurement tactile output) (e.g., asdescribed herein with respect to measurement point 5042, FIG. 5K). Insome embodiments, the electronic device detects movement of themeasurement-point-creation indicator to a second location in therepresentation of the field of view that corresponds to a secondlocation in the three-dimensional space (e.g., as described herein withrespect to FIG. 5M). In some embodiments, while themeasurement-point-creation indicator is over the second location in therepresentation of the field of view, the electronic device detecting asecond touch input on the touch-sensitive display that meets the firstcriteria. In some embodiments, in response to detecting the second touchinput that meets the first criteria, the electronic device adds a secondmeasurement point at the second location in the representation of thefield of view (e.g., as described herein with respect to measurementpoint 5054, FIG. 5N). In some embodiments, in response to detecting thesecond touch input that meets the first criteria, the electronic deviceadds a measurement segment between the first measurement point and thesecond measurement point (e.g., the electronic device adds arepresentation of a measurement by adding a second measurement point anda measurement segment, where the representation of the measurementincludes the first measurement point, the second measurement point, andthe measurement segment) (e.g., as described herein with respect tomeasurement segment 5048, FIG. 5N). In some embodiments, in response todetecting the second touch input that meets the first criteria, theelectronic device generates a second tactile output (e.g., anend-of-measurement tactile output) that is different from the firsttactile output (e.g., the second tactile output differs from the firsttactile output in at least one tactile output property, such asfrequency, amplitude, pattern, duration, etc.) (e.g., as describedherein with respect to tactile output 5056, FIG. 5N). Providingdifferent haptic feedback at the start of a measurement (with just thefirst measurement point) versus when a measurement segment has beencreated helps guide a user while placing measurement points, byindicating where they are in the measurement process. Providing improvedfeedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, as at least one of the one or more cameras moves,the electronic device displays (830) the measurement-point-creationindicator while the representation of the field of view includes aregion corresponding to an identified physical feature in thethree-dimensional space in the field of view of at least one of the oneor more cameras, In addition, the electronic device ceases to displaythe measurement-point-creation indicator while the representation of thefield of view does not include a region corresponding to an identifiedphysical feature in the three-dimensional space (e.g., as describedherein with reference to FIGS. 5E-5G). Displaying or not displaying themeasurement-point-creation indicator, depending on whether a live viewincludes a region that corresponds to an identified physical feature inthe three-dimensional space, provides visual feedback about the presenceor absence of automatically identified features. Providing improvedfeedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 8A-8C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 900, 1000, 1100, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 800 described above withrespect to FIGS. 8A-8C. For example, the inputs, user interface elements(e.g., measurement points, measurement segments, virtual annotations,representations of the physical space or field of view, affordances,alerts, indicators, labels, anchor points, and/or placement userinterface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 800optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 900, 1000, 1100, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 9A-9B are flow diagrams illustrating method 900 of displayinglabels for measurements of a physical space in an augmented realityenvironment in accordance with some embodiments. Method 900 is performedat an electronic device (e.g., portable multifunction device 100 (FIG.1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B)) thatincludes a display (e.g., touch screen 112 (FIG. 1A), display 340 (FIG.3A), or display generation component(s) 304 (FIG. 3B)), an input device(e.g., touch screen 112 (FIG. 1A), touchpad 355 (FIG. 3A), inputdevice(s) 302 (FIG. 3B), or a physical button that is separate from thedisplay), and one or more cameras (e.g., optical sensor(s) 164 (FIG. 1A)or camera(s) 305 (FIG. 3B)), optionally one or more sensors to detectintensities of contacts with a touch-sensitive surface of the inputdevice (e.g., contact intensity sensor(s) 165, FIG. 1A), and optionallyone or more tactile output generators (e.g., tactile output generator(s)163 (FIG. 1A) or tactile output generator(s) 357 (FIG. 3A)). Someoperations in method 900 are, optionally, combined and/or the order ofsome operations is, optionally, changed.

As described below, method 900 provides an intuitive way to providelabels for different measurements, based on the distance between theelectronic device and a given measurement. Providing maximum-size labelsat short distances keeps these labels from getting too big and obscuringlarge portions of the representation of the field of view. Providingminimum-size labels at long distances keeps these labels legible. Andproviding variable-size labels at intermediate distances indicates therelative distances of the corresponding measurements in therepresentation of the field of view. Providing labels for differentmeasurements at different distances in this manner enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to see and use measurement labelswhen operating/interacting with the device).

The electronic device displays (902), on the display, a user interfaceof an application (e.g., an augmented reality measurement application oran application that includes augmented reality measurementfunctionality).

The user interface includes (904) a representation of a field of view ofat least one of the one or more cameras. The representation of the fieldof view is updated over time based on changes to current visual datadetected by at least one of the one or more cameras (e.g., therepresentation of the field of view is a live view). The field of viewincludes a physical object in a three-dimensional space (e.g., a spacein the physical world).

While displaying the representation of the field of view, the electronicdevice detects (906) one or more user inputs (e.g., tap gestures orpress inputs on an affordance which, when activated, adds a measurementpoint to the displayed representation of the field of view), via theinput device, that add, over the representation of the field of view, arepresentation of a first measurement that corresponds to the physicalobject (e.g., a displayed line that corresponds to a measurement of thelength, width, or other dimension of the physical object, where thedisplayed line is superimposed or overlaid on the representation of thefield of view). In some embodiments, the electronic device concurrentlydisplays (908), over the representation of the field of view, therepresentation of the first measurement and a first label that describesthe first measurement. In some embodiments, in accordance with adetermination that a first distance between the electronic device andthe physical object (e.g., a distance from one of the cameras of theelectronic device to the physical object) is less than a first thresholddistance (e.g., a lower distance threshold), the first label is (910)displayed at a first threshold size (e.g., an upper size threshold, or amaximum size, that does not change as the distance between theelectronic device and the physical object decreases below the firstthreshold distance). In some embodiments, in accordance with adetermination that the first distance between the electronic device andthe physical object is greater than a second threshold distance (e.g.,an upper distance threshold) that is greater than the first thresholddistance, the first label is displayed at a second threshold size thatis smaller than the first threshold size (e.g., a lower size threshold,or a minimum size, that does not change as the distance between theelectronic device and the physical object increases above the secondthreshold distance). In some embodiments, in accordance with adetermination that the first distance between the electronic device andthe physical object is between the first threshold distance and thesecond threshold distance, the first label is displayed at a size,between the first threshold size and the second threshold size, thatdepends on the first distance between the electronic device and thephysical object. Variations in label size based on distance between theelectronic device and the physical object are shown in and describedherein with reference to FIGS. 5AX-5BE.

In some embodiments, while concurrently displaying, over therepresentation of the field of view, the representation of themeasurement and the label that describes the measurement: while theelectronic device (or one of the cameras of the electronic device) isthe first distance from the physical object, the electronic devicedisplays (912) one or more first scale markers along the representationof the measurement at a first scale (e.g., displaying the one or morefirst scale markers at intervals of a first predefined distance alongthe representation of the measurement, corresponding to intervals of afirst predefined physical distance in the three-dimensional space). Insome embodiments, the electronic device detects movement of theelectronic device (or movement of the one or more cameras thereof) thatmoves the electronic device to a second distance from the physicalobject. In some embodiments, while the electronic device is the seconddistance from the physical object, the electronic device displays one ormore second scale markers along at least a portion of the representationof the measurement at a second scale that is distinct from the firstscale (e.g., the electronic device displays the one or more second scalemarkers at intervals of a second predefined distance along therepresentation of the measurement, corresponding to intervals of asecond predefined physical distance in the three-dimensional space).Variations in the scale of displayed markers based on distance betweenthe electronic device and the physical object are shown in and describedherein with reference to FIGS. 5AX-5BE.

In some embodiments, the one or more first scale markers at the firstscale are displayed along the representation of the measurement whilethe electronic device is within a first predefined range of distancesfrom the physical object (e.g., distances greater than the secondthreshold distance, or distances between the first threshold distanceand the second threshold distance), where the first predefined range ofdistances includes the first distance. In some embodiments, the one ormore second scale markers at the second scale are displayed along atleast a portion of the representation of the measurement (e.g., if onlya portion of the representation of the measurement continues to bedisplayed in the user interface as a result of the movement of theelectronic device closer to the physical object) while the electronicdevice is within a second predefined range of distances from thephysical object (e.g., distances between the first threshold distanceand the second threshold distance, or distances less than the firstthreshold distance), where the second predefined range of distancesincludes the second distance.

In an example, the detected movement of the electronic device moves theelectronic device closer to the physical object, such that the firstdistance is greater than the second distance. While the electronicdevice is the first distance from the physical object, the one or morescale markers are displayed at intervals along the representation of themeasurement corresponding to intervals of one foot in thethree-dimensional space. In the same example, while the electronicdevice is the second distance from the physical object, the one or morescale markers are displayed at intervals along the representation of themeasurement corresponding to intervals of one inch in thethree-dimensional space. One of ordinary skill in the art will recognizethat different predefined physical distances may be denoted by the scalemarkers displayed while the device is at a respective distance from thephysical object, such as meters, decimeters, centimeters, millimeters,yards, feet, inches, quarter inches, or any other suitable distance.

Providing scale markers that automatically change scale as the distancefrom a measurement to the device changes is more efficient thanrequiring a user to change the scale manually. Performing an operationwhen a set of conditions has been met (e.g., distance conditions)without requiring further user input enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the electronic device detects (914) a second set ofone or more user inputs that add, over the representation of the fieldof view, a representation of a second measurement that corresponds to arespective physical object (e.g., the same physical object to which thefirst measurement corresponds, or a different physical object in thethree-dimensional space) in the three-dimensional space. In someembodiments, the electronic device concurrently displays, over therepresentation of the field of view, the representation of the secondmeasurement and a second label that describes the second measurement. Insome embodiments, in accordance with a determination that a seconddistance between the electronic device and the respective physicalobject is less than the first threshold distance, the second label isdisplayed at the first threshold size. In some embodiments, inaccordance with a determination that the second distance between theelectronic device and the respective physical object is greater than thesecond threshold distance, the second label is displayed at the secondthreshold size. In some embodiments, in accordance with a determinationthat the second distance between the electronic device and therespective physical object is between the first threshold distance andthe second threshold distance, the second label is displayed at a size,between the first threshold size and the second threshold size, thatdepends on the second distance between the electronic device and therespective physical object. In some embodiments, the representation ofthe field of view concurrently displays multiple labels that change insize as the field of view changes.

In some embodiments, the first distance between the electronic deviceand the physical object to which the first measurement corresponds is(916) different from the second distance between the electronic deviceand the respective physical object to which the second measurementcorresponds. In some embodiments, the first label is displayed at afirst size (e.g., based on the first distance), the second label isdisplayed at a second size (e.g., based on the second distance), and thefirst size is different from the second size. In an example, the firstdistance is in a first predefined range of distances, and the seconddistance is in a second predefined range of distances (e.g., the firstpredefined range is one of: distances less than the first thresholddistance, distances greater than the second threshold distance, ordistances between the first threshold distance and the second thresholddistance, and the second predefined range is a different one of theaforementioned ranges). In another example, the first distance and thesecond distances are different distances in the range of distancesbetween the first threshold distance and the second threshold distance,and, accordingly, the respective sizes of their associated labels (e.g.,the labels corresponding to the measurements at the first distance andthe second distance) are different.

It should be understood that the particular order in which theoperations in FIGS. 9A-9B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 1000, 1100, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 900 described above withrespect to FIGS. 9A-9B. For example, the inputs, user interface elements(e.g., measurement points, measurement segments, virtual annotations,representations of the physical space or field of view, affordances,alerts, indicators, labels, anchor points, and/or placement userinterface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 900optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 1000, 1100, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 10A-10B are flow diagrams illustrating method 1000 of measuringand interacting with rectangular areas in a physical space in anaugmented reality environment in accordance with some embodiments.Method 1000 is performed at an electronic device (e.g., portablemultifunction device 100 (FIG. 1A), device 300 (FIG. 3A), or computersystem 301 (FIG. 3B)) that includes a display (e.g., touch screen 112(FIG. 1A), display 340 (FIG. 3A), or display generation component(s) 304(FIG. 3B)), an input device (e.g., touch screen 112 (FIG. 1A), touchpad355 (FIG. 3A), input device(s) 302 (FIG. 3B), or a physical button thatis separate from the display), and one or more cameras (e.g., opticalsensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)), optionally one ormore sensors to detect intensities of contacts with a touch-sensitivesurface of the input device (e.g., contact intensity sensor(s) 165, FIG.1A), and optionally one or more tactile output generators (e.g., tactileoutput generator(s) 163 (FIG. 1A) or tactile output generator(s) 357(FIG. 3A)). Some operations in method 1000 are, optionally, combinedand/or the order of some operations is, optionally, changed.

As described below, method 1000 provides an intuitive way to detect andindicate a rectangle that adjoins a measurement. By having the detectedrectangle adjoin a measurement made in response to user inputs, method1000 reduces the risk that the device will detect and indicaterectangles that are not relevant to the user (e.g., are not beingmeasured), thereby creating a more efficient human-machine interface.For battery-operated electronic devices, enabling faster and moreefficient detection and display of relevant rectangles conserves powerand increases the time between battery charges.

The electronic device displays (1002), on the display, a user interfaceof an application (e.g., an augmented reality measurement application oran application that includes augmented reality measurementfunctionality).

The user interface includes (1004) a representation of a field of viewof at least one of the one or more cameras. The representation of thefield of view is updated over time based on changes to current visualdata detected by at least one of the one or more cameras (e.g., therepresentation of the field of view is a live view). The field of viewincludes a physical object (or portion thereof) in a three-dimensionalspace (e.g., a space in the physical world).

While displaying the representation of the field of view, the electronicdevice detects (1006) one or more user inputs (e.g., tap gestures orpress inputs on an affordance which, when activated, adds a measurementpoint to the displayed representation of the field of view), via theinput device, that add, over the representation of the field of view, arepresentation of a first measurement that corresponds to the physicalobject (e.g., the detected user inputs cause the electronic device toadd a displayed line that corresponds to a measurement of an edge of thephysical object, where the displayed line is superimposed or overlaid onthe representation of the field of view).

The representation of the first measurement includes (1008) a firstendpoint that corresponds to a first location on the physical object.The representation of the first measurement includes a second endpointthat corresponds to a second location on the physical object. Therepresentation of the first measurement includes a first line segmentconnecting the first endpoint and the second endpoint. The addition of arepresentation of a measurement including two endpoints and a first linesegment connecting the two endpoints is shown in shown in and describedherein with reference to FIGS. 5J-5O.

The electronic device determines (1010), based in part on the firstmeasurement (or the representation thereof), a first area in therepresentation of the field of view that adjoins the first line segmentof the first measurement. The first area corresponds to a physicalrectangular area (e.g., the entire physical rectangular area or aportion thereof) in the three-dimensional space. For example, the firstarea corresponds to a physical rectangular area, or a portion thereof,of the physical object and the first measurement corresponds to one edgeof the physical rectangular area of the physical object). In someembodiments, the first area in the representation of the field of viewis not displayed as rectangular in the representation of the field ofview, due to a viewing angle of at least one of the one or more cameraswith respect to the physical rectangular area. In some embodiments, theelectronic device determines that the physical area corresponding to thefirst area in the representation of the field of view is a rectangulararea based on image processing (e.g., using depth estimation) of therepresentation of the field of view. In some embodiments, the field ofview of at least one of the one or more cameras includes a first portionof the physical rectangular area without including one or moreadditional portions of the physical rectangular area. In someembodiments, viewing the one or more additional portions of the physicalrectangular area requires movement of the device that moves the field ofview of at least one of the one or more cameras to include the one ormore additional portions of the physical rectangular area.

The electronic device displays (1012) an indication of the first area inthe user interface (e.g., indicator 5058, FIG. 5P). The indication isoverlaid (e.g., superimposed) on the first area in the representation ofthe field of view.

In some embodiments, the user interface includes (1014) ameasurement-point-creation indicator that is displayed over therepresentation of the field of view (e.g., reticle 5010 in combinationwith focus point 5012, FIG. 5Y). In some embodiments, the indication ofthe first area is displayed in accordance with a determination that themeasurement-point-creation indicator is displayed over the first area inthe representation of the field of view of the one or more cameras(e.g., as shown in and described herein with reference to FIG. 5Y). Insome embodiments, while the measurement-point-creation indicator (or atleast a portion thereof) is over the first area, the electronic devicedetects a user input via the input device (e.g., a tap gesture or pressinput on a touch-sensitive surface at a location corresponding to anaffordance which, when activated, adds a measurement or measurementpoint to the displayed representation of the field of view). In someembodiments, in response to detecting the user input while themeasurement-point-creation indicator is over the first area, theelectronic device changes a visual appearance of the indication of thefirst area to indicate that the first area has been confirmed (e.g., asshown in and described herein with reference to FIG. 5Z).

In some embodiments, the user input is detected while themeasurement-point-creation indicator is over the first area and whilethe indication of the first area is displayed, and the visual appearanceof the indication of the first area is changed in response to (e.g., inaccordance with) detecting the user input both while themeasurement-point-creation indicator is over the first area and whilethe indication of the first area is displayed. In some embodiments, themethod includes displaying, over the representation of the field ofview, one or more labels that describe the first area (e.g.,concurrently displayed with the indication displayed with changed visualappearance). In some embodiments, the one or more labels that describethe first area include a label that indicates a length of a first sideof the first area (e.g., a length of the physical rectangular area), alabel that indicates a length of a second side of the first area (e.g.,a width of the physical rectangular area), and/or a label that indicatesan area of the first area (e.g., an area of the physical rectangulararea) (e.g., as shown in and described herein with reference to FIG.5Z).

Changing the appearance of the indication of the first area providesvisual feedback that the electronic device has detected the user'sconfirmation that the rectangle adjoining the first measurement iscorrect. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the one or more user inputs add (1016), over therepresentation of the field of view, a representation of a secondmeasurement that corresponds to the physical object (e.g., the detecteduser inputs cause the electronic device to add a second displayed linethat corresponds to a measurement of a second edge of the physicalobject, where the second displayed line is superimposed or overlaid onthe representation of the field of view) (e.g., a measurementrepresented by measurement segment 5066 and its corresponding endpoints,FIG. 5Y). In some embodiments, the representation of the secondmeasurement includes the second endpoint that corresponds to the secondlocation on the physical object (e.g., measurement point 5054, FIG. 5Y).In some embodiments, the representation of the second measurementincludes a third endpoint that corresponds to a third location on thephysical object (e.g., the third location is different from the firstlocation and the second location) (e.g., measurement point 5090, FIG.5Y). In some embodiments, the representation of the second measurementincludes a second line segment connecting the second endpoint and thethird endpoint (e.g., measurement segment 5066, FIG. 5Y). In someembodiments, the first area in the representation of the field of viewof the one or more cameras is determined based on the first measurementand the second measurement. In some embodiments, the first area adjoinsthe first line segment of the first measurement and the second linesegment of the second measurement. Determining the first area based ontwo measurements reduces the risk that the device will detect andindicate rectangles that are not relevant to the user (e.g., are notbeing measured), thereby enhancing the operability of the device andmaking the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, the field of view of at least one of the one ormore cameras includes (1018) a first portion of the physical rectangulararea, and the first area corresponds to the first portion of thephysical rectangular area (e.g., as described with reference to theregion indicated by indicator 5104, FIG. 5AB). In some embodiments, theelectronic device detects movement of the electronic device that movesthe field of view of at least one of the one or more cameras (e.g., asshown in and described herein with reference to FIGS. 5AC-5AD). In someembodiments, in response to detecting the movement of the electronicdevice that moves the field of view, the electronic device updates therepresentation of the field of view over time to display one or moreindications of one or more additional areas that correspond to one ormore additional portions of the physical rectangular area. In someembodiments, in accordance with a determination that an aggregate area,including the first area and the one or more additional areas displayedover time, corresponds to the entire physical rectangular area, theelectronic device displays, over the representation of the field ofview, a label that describes a measurement that corresponds to thephysical rectangular area (e.g., as described herein with reference tolabel 5110, FIG. 5AE). In some embodiments, the label indicates an areaof the measurement (e.g., an area of the entire physical rectangulararea). Automatically showing, as the field of view changes, indicationsof additional areas that correspond to the physical rectangular areaprovides visual feedback that the electronic device has correctlydetected the physical rectangular area. Providing improved feedbackenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, after displaying, over the representation of thefield of view, the label that describes the measurement that correspondsto the physical rectangular area in accordance with the determinationthat the aggregate area corresponds to the entire physical rectangulararea, the electronic device detects (1020) further movement of theelectronic device that moves the field of view of at least one of theone or more cameras such that the representation of the field of viewincludes the first area corresponding to the first portion of thephysical rectangular area. In some embodiments, in accordance with thedetermination that the aggregate area corresponds to the entire physicalrectangular area, the electronic device displays, over the first areacorresponding to the first portion of the physical rectangular area, thelabel that describes the measurement that corresponds to the physicalrectangular area (e.g., as described herein with reference to FIG. 5AF).Displaying the label describing the measurement of the physicalrectangular area at different portions of the physical rectangular areawhen those portions are (re)displayed provides visual feedback that theelectronic device has correctly detected the physical rectangular areaand that the measurement has been correctly associated with all portionsof the physical rectangular area. Providing improved feedback enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 10A-10B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 900, 1100, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 1000 described above withrespect to FIGS. 10A-10B. For example, the inputs, user interfaceelements (e.g., measurement points, measurement segments, virtualannotations, representations of the physical space or field of view,affordances, alerts, indicators, labels, anchor points, and/or placementuser interface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 1000optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 900, 1100, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 11A-11B are flow diagrams illustrating method 1100 of interactingwith and managing measurement information in an augmented realityenvironment in accordance with some embodiments. Method 1100 isperformed at an electronic device (e.g., portable multifunction device100 (FIG. 1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B))that includes a touch-sensitive display (e.g., touch screen 112 (FIG.1A), or display generation component(s) 304 in combination with inputdevice(s) 302 (FIG. 3B)), and one or more cameras (e.g., opticalsensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)), optionally one ormore sensors to detect intensities of contacts with the touch-sensitivedisplay (e.g., contact intensity sensor(s) 165, FIG. 1A), and optionallyone or more tactile output generators (e.g., tactile output generator(s)163 (FIG. 1A) or tactile output generator(s) 357 (FIG. 3A)). Someoperations in method 1100 are, optionally, combined and/or the order ofsome operations is, optionally, changed.

As described below, method 1100 provides an intuitive way to shareinformation about a measurement, by initiating a process for sharing theinformation in response to detecting a touch input on a representationof the measurement. Method 1100 reduces the number, extent, and/ornature of the inputs from a user when sharing information about ameasurement, thereby creating a more efficient human-machine interface.For battery-operated electronic devices, enabling a user to shareinformation about a measurement faster and more efficiently conservespower and increases the time between battery charges.

The electronic device displays (1102), on the touch-sensitive display, afirst user interface of an application (e.g., an augmented realitymeasurement application or an application that includes augmentedreality measurement functionality).

The first user interface includes (1104) a representation of a field ofview of at least one of the one or more cameras. The representation ofthe field of view is updated over time based on changes to currentvisual data detected by at least one of the one or more cameras (e.g.,the representation of the field of view is a live view). The field ofview includes a physical object in a three-dimensional space (e.g., aspace in the physical world). A representation of a measurement of thephysical object is superimposed on an image of the physical object inthe representation of the field of view.

While displaying the first user interface, the electronic device detects(1106) a first touch input on the touch-sensitive display on therepresentation of the measurement (e.g., a tap, double tap, or pressinput on the displayed measurement) (e.g., touch-input 5182, FIG. 5BL).

In response to detecting the first touch input on the touch-sensitivedisplay on the representation of the measurement, the electronic deviceinitiates (1108) a process for sharing information about the measurement(e.g., sending information about the measurement to a clipboard processor a communication application (e.g., a text messaging application, ane-mail application, a file transfer application), etc.) (e.g., asdescribed herein with reference to FIGS. 5BL-5BM). In some embodiments,the process includes adding the measurement information to a clipboard.In some embodiments, the process includes sending the information to asecond application. In some embodiments, initiating the process includesdisplaying a second user interface that includes user-selectable optionsfor sharing the information about the measurement, such as a share sheetuser interface. In some embodiments, the second user interface includesthe information describing the measurement. In some embodiments, theinformation includes an automatically generated semantic labelclassifying the physical object (e.g., as a window, wall, floor, ortable) on which the measurement is superimposed. In some embodiments,the information includes an automatically generated semantic labelclassifying a relationship between the first measurement and thephysical object (e.g., a length, width, height, or depth of the physicalobject).

In some embodiments, initiating the process for sharing informationabout the measurement includes (1110) copying the information about themeasurement (e.g., to a clipboard process provided by an operatingsystem of the electronic device). In some embodiments, after copying theinformation about the measurement, the electronic device detects one ormore user inputs to paste the information about the measurement to adestination on the electronic device. In some embodiments, in responseto detecting the one or more user inputs to paste the information aboutthe measurement to the destination on the electronic device, theelectronic device displays the information about the measurement at thedestination on the electronic device. Enabling copying and pasting ofthe information about the measurement makes it easy to share themeasurement within the same application and with other applications onthe electronic device. Making information available to multipleapplications enhances the operability of the device and makes theuser-device interface more efficient (e.g., by making it easy to selectan application for sharing or sending the measurement).

In some embodiments, initiating the process for sharing the informationabout the measurement includes (1112) displaying a second user interface(e.g., measurement management interface 5184, FIG. 5BM) that includesone or more activatable user interface elements, where a respectiveactivatable user interface element in the one or more activatable userinterface elements corresponds to a respective destination for theinformation about the measurement (e.g., icons 5192, 5194, and 5196,FIG. 5BM). In some embodiments, each of the one or more activatable userinterface elements corresponds to a respective application (other thanthe first application) or process on the electronic device (e.g., amessaging application (e.g., icon 5194, FIG. 5BM), an email application(e.g., icon 5192, FIG. 5BM), a notetaking application, a file transferprotocol (e.g., icon 5196, FIG. 5BM), etc.). In some embodiments, theelectronic device detects a second touch input on the touch-sensitivedisplay on a respective activatable user interface element in the seconduser interface. In some embodiments, in response to detecting the secondtouch input, the electronic device transmits the information about themeasurement to the respective destination corresponding to therespective activatable user interface element.

In some embodiments, transmitting the information about the measurementto the respective destination includes transmitting the information to asecond application (e.g., a notetaking application) on the electronicdevice. In some embodiments, transmitting the information about themeasurement to the respective destination includes displaying a thirduser interface for reviewing, editing, and/or annotating the informationabout the measurement prior to transmitting the information to asubsequent destination (e.g., the third user interface includes anaffordance upon selection of which the information, including any editsand annotations, is transmitted from the respective destination (e.g., amessaging application or an email application) to a subsequentdestination (e.g., another electronic device)). In some embodiments,transmitting the information about the measurement to the respectivedestination includes transmitting the information to a second electronicdevice via a file transfer protocol between the electronic device andthe second electronic device.

Providing a user interface with activatable user interface elements formultiple destinations for the shared measurement (e.g., a share sheetwith destination icons) makes it easy to share the measurement withthese destinations. Providing multiple sharing destination optionsenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by making it easy to select anapplication for sharing or sending the measurement).

In some embodiments, in response to detecting the first touch input onthe touch-sensitive display on the representation of the measurement,the electronic device displays (1114) the information about themeasurement. In some embodiments, the information about the measurementincludes a magnitude of the measurement (e.g., as shown in measurementmanagement interface 5184, FIG. 5BM). In some embodiments, theinformation about the measurement includes a semantic label classifyinga relationship between the measurement and the physical object (e.g., alength, width, height, or depth of the physical object) (e.g., asdescribed herein with reference to label 5186-b in measurementmanagement interface 5184, FIG. 5BM). In some embodiments, theinformation about the measurement includes a label (e.g., a text label)classifying a relationship between the measurement and an identifiedanchor feature of the three-dimensional space (e.g., whether an area inthe representation of the field of view corresponds to a physicalrectangular area that is parallel to the ground). In some embodiments,where the measurement corresponds to a physical rectangular area andincludes a length measurement, a width measurement, and an areameasurement, the length, width, and area are displayed, and the lengthand width are displayed more prominently than the area. Providinginformation about the measurement, in addition to initiating a processfor sharing the information, in response to detecting a touch input on arepresentation of the measurement, provides visual feedback that theuser has selected the correct measurement for sharing, and allows a userto see and use the information. Providing improved feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, the displaying of the information about themeasurement is (1116) performed in accordance with a determination thatthe first touch input meets first criteria, where the first criteriainclude a requirement that an intensity of a contact in the first touchinput meet (e.g., reach or exceed) a respective intensity threshold inorder for the first criteria to be met (e.g., as described herein withreference to FIGS. 5BL-5BM). In some embodiments, the respectiveintensity threshold is a light-press intensity threshold IT_(L) that isabove a contact detection intensity threshold IT₀. Providing informationabout the measurement in response to detecting a touch input on arepresentation of the measurement that meets intensity criteria reducesaccidental, unwanted display of the information. Performing an operationwhen intensity criteria have been met enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the electronic device determines (1118) aclassification of the physical object (e.g., a classification of thephysical object as a respective structural feature (such as a window, awall, a floor, etc.) or as a respective piece of furniture or fixture(such as a table)). In some embodiments, the information about themeasurement includes a label indicating the classification of thephysical object (e.g., as described herein with reference to label5186-a in measurement management interface 5184, FIG. 5BM). In someembodiments, the electronic device classifies the physical object basedon image processing (e.g., using feature recognition) of therepresentation of the field of view of the one or more cameras.Automatically classifying and labeling the physical object, withoutrequiring further user input, enhances the operability of the device andmakes the user-device interface more efficient by reducing (oreliminating) the need for the user to manually classify and labelphysical objects.

In some embodiments, the representation of the measurement was (1120)added to the user interface of the application based at least in part onmovement of the electronic device in a first direction during themeasurement. In some embodiments, the electronic device determines aclassification of a relationship between the measurement and thephysical object (e.g., a classification of the measurement ascorresponding to a length, width, height, or depth of the physicalobject) based at least in part on the movement of the electronic devicein the first direction during the measurement (e.g., as described hereinwith reference to FIG. 5BM).

In some embodiments, the information about the measurement includes alabel indicating the classification of the relationship between themeasurement and the physical object. For example, the electronic deviceclassifies the measurement as a height of the physical object based onthe movement of the electronic device being in a vertical directionduring the measurement, or as a width of the physical object based onthe movement of the electronic device being in a horizontal directionduring the measurement. In some embodiments, the electronic deviceclassifies the relationship between the measurement and the physicalobject based further on image processing (e.g., feature recognition) ofthe representation of the field of view to determine respectivedistances between the electronic device and respective points on thephysical object corresponding to respective points along themeasurement. For example, the electronic device classifies themeasurement as a depth of the physical object based further on adetermination that a first point on the physical object, correspondingto a first endpoint of the measurement, is further from (or closer to)(e.g., in a z-direction) the electronic device than a second point onthe physical object, corresponding to a second endpoint of themeasurement.

Automatically classifying and labeling a measurement, based in part onthe movement of the device during the measurement, without requiringfurther user input, enhances the operability of the device and makes theuser-device interface more efficient, by reducing (or eliminating) theneed for the user to manually classify and label measurements.

It should be understood that the particular order in which theoperations in FIGS. 11A-11B have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 900, 1000, 1200, 1300, and 1400) are alsoapplicable in an analogous manner to method 1100 described above withrespect to FIGS. 11A-11B. For example, the inputs, user interfaceelements (e.g., measurement points, measurement segments, virtualannotations, representations of the physical space or field of view,affordances, alerts, indicators, labels, anchor points, and/or placementuser interface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 1100optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 900, 1000, 1200, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 12A-12C are flow diagrams illustrating method 1200 of providingautomatically determined alignment guides in an augmented realityenvironment in accordance with some embodiments. Method 1200 isperformed at an electronic device (e.g., portable multifunction device100 (FIG. 1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B))that includes a display (e.g., touch screen 112 (FIG. 1A), display 340(FIG. 3A), or display generation component(s) 304 (FIG. 3B)), an inputdevice (e.g., touch screen 112 (FIG. 1A), touchpad 355 (FIG. 3A), orinput device(s) 302 (FIG. 3B), or a physical button that is separatefrom the display), and one or more cameras (e.g., optical sensor(s) 164(FIG. 1A) or camera(s) 305 (FIG. 3B)), optionally one or more sensors todetect intensities of contacts with a touch-sensitive surface of theinput device (e.g., contact intensity sensor(s) 165, FIG. 1A), andoptionally one or more tactile output generators (e.g., tactile outputgenerator(s) 163 (FIG. 1A) or tactile output generator(s) 357 (FIG.3A)). Some operations in method 1200 are, optionally, combined and/orthe order of some operations is, optionally, changed.

As described below, method 1200 provides an intuitive way to provide(virtual) guides in an augmented reality-based measurement, with theguides extending along a direction of movement of a field of view of acamera. Providing measurement guides helps a user position and place(virtual) measurement points quickly and accurately. By automaticallyproviding guides along a direction of movement of a field of view of acamera, method 1200 reduces the number, extent, and/or nature of theinputs from a user when making measurements, thereby creating a moreefficient human-machine interface. For battery-operated electronicdevices, enabling a user to make measurements faster and moreefficiently conserves power and increases the time between batterycharges.

The electronic device displays (1202), on the display, a user interfaceof an application (e.g., an augmented reality measurement application oran application that includes augmented reality measurementfunctionality). The user interface includes (1204) a representation of afield of view of at least one of the one or more cameras; therepresentation of the field of view is updated over time based onchanges to current visual data detected by at least one of the one ormore cameras (e.g., the representation of the field of view is a liveview); and the field of view includes at least a portion of athree-dimensional space (e.g., a space in the physical world thatincludes physical objects).

The electronic device detects (1206) movement of the electronic devicethat moves the field of view of at least one of the one or more camerasin a first direction (e.g., horizontally, or vertically) (or, in someembodiments, in substantially the first direction (e.g., a directionthat is within a predefined threshold angle of a first direction, suchas within 10, 15, 20 or 25 degrees of the first direction)).

While detecting the movement of the electronic device that moves thefield of view in the first direction (1208), the electronic deviceupdates (1210) the representation of the field of view in accordancewith the movement of the electronic device; identifies (1212) one ormore first elements (or features) in the representation of the field ofview that extend along the first direction (e.g., a detected edge, adetected plane, etc.); and, based at least in part on the determinationof the one or more first elements, displays (1214), in therepresentation of the field of view, a first guide that extends in thefirst direction and that corresponds to one of the one or more firstidentified elements (e.g., as described herein with reference to virtualguide 5050, FIG. 5M). In some embodiments, the electronic devicedisplays a plurality of guides in the first direction (e.g., asdescribed herein with reference to virtual guides 5106, FIG. 5AB). Insome embodiments, each of the plurality of guides corresponds to arespective identified element that extends along the first direction(e.g., a respective edge, of a respective physical object in thethree-dimensional space, that extends along the first direction). Insome embodiments, the electronic device displays one or more guides inthe first direction while detecting the movement of the electronicdevice.

In some embodiments, the field of view includes (1216) a plurality ofelements, and the plurality of elements includes one or more elementsthat extend in directions other than the first direction (e.g.,directions perpendicular to or substantially perpendicular to (e.g.,within a predefined threshold angle of being perpendicular to) the firstdirection, or directions that are greater than a predefined thresholdangle from the first direction, such as greater than 10, 15, 20 or 25degrees from the first direction). In some embodiments, while detectingthe movement of the electronic device that moves the field of view inthe first direction, the electronic device forgoes displaying guidesthat extend in directions other than the first direction (e.g., asdescribed herein with reference to FIGS. 5AN-5AO and 5AV). For example,the electronic device determines an axis of a physical object (e.g., atable or a wall) along which to extend guides based on the direction ofmovement of the camera. In some embodiments, the electronic devicedisplays one or more guides, corresponding to one or more elements inthe field of view, that extend in the direction of movement of at leastone camera, without displaying guides that extend in directions otherthan the direction of motion of the at least one camera. Providing oneor more guides along the direction of movement of the field of view,without providing guides that extend in other directions, avoidsdisplaying guides that are not likely to be relevant to the measurementbeing made. Reducing clutter in the user interface enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, prior to detecting the movement of the electronicdevice that moves the field of view in the first direction (1218), theelectronic device detects a first touch input on the touch-sensitivedisplay, and, in response to detecting the first touch input, adds anddisplays a first (virtual) measurement point at a first location in therepresentation of the field of view that corresponds to a first locationin the three-dimensional space (e.g., measurement point 5054, FIG. 5R).Displaying the first guide is further based on a determination that theone or more first elements correspond to the first location in thethree-dimensional space (e.g., the first location is a point along adetected edge). In addition, the first guide includes the first locationin the representation of the field of view (e.g., the guide overlapswith the measurement point, or extends from the measurement point in thefirst direction) (e.g., as described herein with reference to virtualguide 5070, FIG. 5S). Providing a guide that extends from and/orincludes a first measurement point along the direction of movement ofthe field of view helps a user to place a second measurement point, andthereby make a measurement. Providing this visual feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper measurementinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, the first measurement point at the first locationin the representation of the field of view is (1220) amost-recently-added measurement point in the representation of the fieldof view. Providing a guide that extends from and/or includes themost-recently-added measurement point along the direction of movement ofthe field of view helps a user to place the next measurement point, andthereby make a measurement between the two most-recently-addedmeasurement points. Providing this visual feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper measurementinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, after detecting the movement of the electronicdevice that moves the field of view in the first direction (e.g.,horizontally, as described herein with reference to FIG. 5S), theelectronic device (1222) detects movement of the electronic device thatmoves the field of view in a second direction (e.g., vertically, asdescribed herein with reference to FIG. 5T). In response to detectingthe movement of the electronic device that moves the field of view inthe second direction, the electronic device ceases to display the firstguide that extends in the first direction. In some embodiments, theelectronic device ceases to display any guides in any directions otherthan the second direction or within a predefined threshold angle of thesecond direction (e.g., as described herein with reference to FIG. 5T).While detecting the movement of the electronic device that moves thefield of view in the second direction, the electronic device updates therepresentation of the field of view in accordance with the movement ofthe electronic device, identifies one or more second elements in therepresentation of the field of view that extend along the seconddirection, and, based at least in part on the determination of the oneor more second elements, displays, in the representation of the field ofview, a second guide that extends in the second direction and thatcorresponds to one of the one or more identified second elements (e.g.,virtual guide 5072, FIG. 5T). Automatically changing guides, as thedirection of movement of the field of view changes, displays guides thatare more likely to be relevant to the measurement being made (and ceasesto display guides that are less likely to be relevant to the measurementbeing made). Reducing clutter in the user interface enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, at least one of the one or more first identifiedelements to which the first guide corresponds is (1224) an elementidentified as an edge of a physical object in the three-dimensionalspace (e.g., as described herein with reference to FIG. 5S).Automatically displaying a guide along an edge of a physical object thatruns in the first direction helps a user to place measurement points andmake measurements along the edge. Automatically providing a guide alongan edge of an object, without requiring further user input, enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper measurementinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, at least one of the one or more first identifiedelements to which the first guide corresponds is (1226) an elementidentified as a plane in the three-dimensional space (e.g.,corresponding to a surface of a physical object in the three-dimensionalspace). Automatically displaying a guide along a plane that runs in thefirst direction helps a user to place measurement points and makemeasurements along the plane. Automatically providing a guide along aplane, without requiring further user input, enhances the operability ofthe device and makes the user-device interface more efficient (e.g., byhelping the user to provide proper measurement inputs and reducing usermistakes when operating/interacting with the device).

In some embodiments, the user interface includes (1228) ameasurement-point-creation indicator that is displayed over therepresentation of the field of view. In some embodiments, the electronicdevice displays a respective measurement point at a respective locationin the representation of the field of view that corresponds to arespective location in the three-dimensional space. In some embodiments,the electronic device detects movement of the electronic device thatmoves the measurement-point-creation indicator over the respectivemeasurement point in the representation of the field of view, and, inresponse to detecting the movement of the electronic device, while themeasurement-point-creation indicator is displayed over the respectivemeasurement point, the electronic device displays a plurality of guides.A first guide of the plurality of guides is perpendicular to a secondguide of the plurality of guides, and the plurality of guides intersectat the respective measurement point. Display of perpendicular guides isdescribed herein with reference to FIG. 5AT.

In some embodiments, a respective guide of the plurality of guides isdisplayed based at least in part on a determination that the respectiveguide extends along an element in the representation of the field ofview that extends from the measurement point. For example, if themeasurement point corresponds to a corner of a physical object in thethree-dimensional space, the one or more guides include a guide thatextends in a first direction (e.g., along an x-axis) from the corner ofthe physical object along an edge of the physical object that extends inthe first direction (e.g., an edge corresponding to a (horizontal)length of the physical object), as displayed in the representation ofthe field of view. In some embodiments, the one or more guides include aguide that extends in a second direction (e.g., along a y-axis) from thecorner of the physical object along an edge of the physical object thatextends in the second direction (e.g., corresponding to a (vertical)height of the physical object), as displayed in the representation ofthe field of view. In some embodiments, the one or more guides include aguide that extends in a third direction (e.g., along a z-axis) from thecorner of the physical object along an edge of the physical object thatextends in the third direction (e.g., an edge corresponding to a depthof the physical object), as displayed in the representation of the fieldof view.

Automatically displaying a plurality of perpendicular guides at ameasurement point, while a measurement-point-creation indicator isdisplayed over the measurement point, helps a user to place additionalmeasurement points. Automatically providing perpendicular guides at ameasurement point, without requiring further user input, enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper measurementinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, the user interface includes (1230) ameasurement-point-creation indicator that is displayed over a secondlocation in the representation of the field of view. In someembodiments, while displaying the first guide that extends in the firstdirection, the electronic device detects a second touch input on thetouch-sensitive display. In response to detecting the second touchinput, and in accordance with a determination that a distance (e.g., ashortest distance) between the second location and the first guide iswithin a threshold distance, the electronic device adds and displays asecond (virtual) measurement point at the location on the first guidethat is the distance from the second location. For example, FIG. 5Xillustrates the addition of measurement point 5090 at a point on virtualguide 5072. In response to detecting the second touch input, and inaccordance with a determination that a distance between the secondlocation and the first guide is not within the threshold distance, theelectronic device adds and displays the second measurement point at thesecond location (e.g., measurement point 5080 is not within thethreshold distance of virtual guide 5072, and thus measurement point5080 is not added to a point on virtual guide 5072). Automaticallyadding a measurement point on a guide (e.g., snapping the measurementpoint to a location on the guide) or adding the measurement point offthe guide, depending on the distance between the location of ameasurement-point-creation indicator and the guide, helps a user toplace measurement points quickly and accurately. Performing an operationwhen a set of conditions has been met (e.g., a distance condition),without requiring further user input, enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper measurement inputs and reducing usermistakes when operating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 12A-12C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 900, 1000, 1100, 1300, and 1400) are alsoapplicable in an analogous manner to method 1200 described above withrespect to FIGS. 12A-12C. For example, the inputs, user interfaceelements (e.g., measurement points, measurement segments, virtualannotations, representations of the physical space or field of view,affordances, alerts, indicators, labels, anchor points, and/or placementuser interface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 1200optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 900, 1000, 1100, 1300, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 13A-13C are flow diagrams illustrating method 1300 ofautomatically removing previously-added virtual annotations inaccordance with some embodiments. Method 1300 is performed at anelectronic device (e.g., portable multifunction device 100 (FIG. 1A),device 300 (FIG. 3A), or computer system 301 (FIG. 3B)) that includesone or more input devices (e.g., touch screen 112 (FIG. 1A), touchpad355 (FIG. 3A), or input device(s) 302 (FIG. 3B), or a physical buttonthat is separate from the display), one or more display devices (e.g.,touch screen 112 (FIG. 1A), display 340 (FIG. 3A), or display generationcomponent(s) 304 (FIG. 3B)), and one or more cameras (e.g., opticalsensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)), optionally one ormore sensors to detect intensities of contacts with a touch-sensitivesurface of the input device (e.g., contact intensity sensor(s) 165, FIG.1A), and optionally one or more tactile output generators (e.g., tactileoutput generator(s) 163 (FIG. 1A) or tactile output generator(s) 357(FIG. 3A)). Some operations in method 1300 are, optionally, combinedand/or the order of some operations is, optionally, changed.

As described below, method 1300 provides an intuitive way toautomatically delete prior virtual annotations (such as prioraugmented-reality-based measurements) that are not connected or relatedto current virtual annotations (such as current augmented-reality-basedmeasurements). Automatically deleting prior, unrelated virtualannotations prevents the augmented reality user interface from becomingcluttered with earlier virtual annotations while making new virtualannotations. As described herein, method 1300 makes it easy to makeindividual virtual annotations without having to manually delete priorvirtual annotations, and also makes it easy to take a series of relatedvirtual annotations (e.g., connected virtual annotations) withoutdeleting early virtual annotations in the series as new virtualannotations are added to the series. The method reduces the number,extent, and/or nature of the inputs from a user when makingaugmented-reality-based virtual annotations, thereby creating a moreefficient human-machine interface. For battery-operated electronicdevices, enabling a user to make augmented-reality-based virtualannotations faster and more efficiently conserves power and increasesthe time between battery charges.

In particular, when the virtual annotations are augmented-reality-basedmeasurements, automatically deleting prior, unrelatedaugmented-reality-based measurements prevents the measurement userinterface from becoming cluttered with earlier measurements while makingnew measurements. These automatic deletions also remove measurements forwhich the electronic device may no longer have accurate mappings to thephysical space. As described herein, method 1300 makes it easy to takeindividual measurements without having to manually delete priormeasurements, and also makes it easy to take a series of relatedmeasurements (e.g., connected measurements) without deleting earlymeasurements in the series as new measurements are added to the series.The method reduces the number, extent, and/or nature of the inputs froma user when making augmented-reality-based measurements, therebycreating a more efficient human-machine interface. For battery-operatedelectronic devices, enabling a user to make augmented-reality-basedmeasurements faster and more efficiently conserves power and increasesthe time between battery charges.

The electronic device displays (1302), via the one or more displaydevices, a user interface that includes a representation of a physicalspace (e.g., a live preview of a portion of the physical space that isin the field of view of at least one of the one or more cameras). Forexample, user interface 5006 in FIG. 5BS includes a representation ofphysical space 5000.

While displaying the representation of the physical space, theelectronic device receives (1304) a first set of one or more inputs tocreate a virtual annotation (e.g., a shape, line, rectangle,measurement, or the like) in the representation of the physical space(e.g., an input to drop a point followed by movement of the electronicdevice relative to the physical space followed by another input to dropa point).

In response to receiving the first set of one or more inputs, theelectronic device adds (1306) a first virtual annotation to therepresentation of the physical space. The first virtual annotation islinked to a portion of the representation of the physical space. Forexample, as shown in FIGS. 5BS-5BU, a virtual annotation that includesmeasurement segment 5212 and its associated endpoints 5206 and 5216 iscreated in user interface 5006 and linked to a representation of(physical) object 5202. As another example, as shown in FIGS. 5BV-5BY, avirtual annotation that includes measurement segment 5222 and itsassociated endpoints 5220 and 5224 is created in user interface 5006 andlinked to a representation of object 5202.

In some embodiments, adding a virtual annotation to the representationof the physical space includes creating a virtual annotation and linkingit to a position in the representation of the physical space, so thatthe virtual annotation appears fixed or substantially fixed in therepresentation of the physical space. In some embodiments, a virtualannotation “in” the representation of the physical space or “added to”the representation of the physical space is actually added to a model ofthe physical space and is drawn on top of camera images of the physicalspace when the portion of the representation of the physical space towhich the virtual annotation is linked appears in the camera images ofthe physical space, to give the impression that the virtual annotationis in the physical space. In some embodiments, a virtual annotation“removed from” the representation of the physical space is actuallyremoved from a model of the physical space and, once it has been“removed from” the representation of the physical space, it is no longerdrawn on top of camera images of the physical space when the portion ofthe representation of the physical space to which the virtual annotationwas linked appears in the camera images of the physical space, to givethe impression that the virtual annotation is no longer in the physicalspace.

After adding the first virtual annotation to the representation of thephysical space, the electronic device receives (1310) a second set ofone or more inputs associated with the representation of the physicalspace.

In response to receiving the second set of one or more inputs associatedwith the representation of the physical space (1312), in accordance witha determination that the second set of one or more inputs corresponds toa request to create a virtual annotation in the representation of thephysical space that is within a threshold distance from the firstvirtual annotation (1314), the electronic device creates a secondvirtual annotation in the representation of the physical space (e.g.,linked to a second portion of the representation of the physical space)while maintaining the first virtual annotation in the representation ofthe physical space. In some embodiments, the first virtual annotationand the second virtual annotation are concurrently displayed. In someembodiments, the threshold distance is zero (e.g., the second virtualannotation must be connected with the first virtual annotation in orderfor the first virtual annotation to be maintained when the secondvirtual annotation is created). In some embodiments, the thresholddistance is greater than zero (e.g., if the second virtual annotation iswithin a predetermined distance to the first virtual annotation, thefirst virtual annotation is maintained when the second virtualannotation is created, even if the second virtual annotation is notconnected to or touching the first virtual annotation). For example, asshown in FIGS. 5BZ-5CF, because measurement segment 5232 is within athreshold distance from previously-added measurement segment 5222,previously-added measurement segment 5222 is maintained in therepresentation of physical space 5000 in user interface 5006 when/aftermeasurement segment 5232 is created. In another example, as shown inFIGS. 5CG-5CK, because measurement segment 5242 is within a thresholddistance from previously-added measurement segments 5232 and 5222,previously-added measurement segments 5232 and 5222 are maintained inthe representation of physical space 5000 in user interface 5006when/after measurement segment 5242 is created.

In addition, in response to receiving the second set of one or moreinputs associated with the representation of the physical space (1312),in accordance with a determination that the second set of one or moreinputs corresponds to a request to create a virtual annotation in therepresentation of the physical space that is outside of the thresholddistance from the first virtual annotation (1316), the electronic devicecreates a second virtual annotation in the representation of thephysical space (e.g., linked to a second portion of the representationof the physical space) and removes the first virtual annotation from therepresentation of the physical space. In some embodiments, the firstvirtual annotation is removed without an explicit request to remove thefirst virtual annotation (e.g., the device does not detect a user input,separate from the second set of inputs used to create the second virtualannotation, requesting deletion of the first virtual annotation). Forexample, as shown in FIGS. 5BV-5BY, because measurement segment 5222 isoutside of the threshold distance from previously-added measurementsegment 5212, previously-added measurement segment 5212 is removedwhen/after measurement segment 5222 is added.

In some embodiments, creating the second virtual annotation in therepresentation of the physical space while maintaining the first virtualannotation in the representation of the physical space includes (1318)starting to create the second virtual annotation at a location thatcorresponds to at least a portion of the first virtual annotation (e.g.,if the creation of measurement segment 5232 in FIGS. 5CA-5CF had insteadstarted with the placement of measurement point 5234 in FIG. 5CF andended with the placement of measurement point 5228 in FIG. 5CA).Automatically keeping a prior virtual annotation when a subsequentvirtual annotation starts on the prior virtual annotation enables theelectronic device to make a series of related virtual annotations (e.g.,a series of connected measurements), without deleting early annotations(e.g., measurements) in the series as new annotations are added to theseries. Performing an operation when a set of conditions has been met(e.g., when a subsequent virtual annotation starts on the prior virtualannotation) without requiring further user input enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, creating the second virtual annotation in therepresentation of the physical space while maintaining the first virtualannotation in the representation of the physical space includes (1320)completing creation of the second virtual annotation at a location thatcorresponds to at least a portion of the first virtual annotation. Forexample, as shown in FIGS. 5BZ-5CF, creation of measurement segment 5232is completed at a location that corresponds to an endpoint ofmeasurement segment 5222, and measurement segment 5222 is thusmaintained in the representation of physical space 5000. Automaticallykeeping a prior virtual annotation when a subsequent virtual annotationends on the prior virtual annotation enables the electronic device tomake a series of related virtual annotations (e.g., a series ofconnected measurements), without deleting early annotations (e.g.,measurements) in the series as new annotations are added to the series.Performing an operation when a set of conditions has been met (e.g.,when a subsequent virtual annotation ends on the prior virtualannotation) without requiring further user input enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, in response to receiving the second set of one ormore inputs associated with the representation of the physical space(1322), in accordance with a determination that the second set of one ormore inputs correspond to a request to shift a field of view of at leastone of the one or more cameras by more than a threshold amount (e.g.,shifting a field of view of at least one of the one or more cameras byan amount that causes a fidelity of tracking of the link between thefirst virtual annotation and the representation of the physical space todegrade by more than a threshold amount), the electronic device removesthe first virtual annotation from the representation of the physicalspace. In some embodiments, the first virtual annotation is removedwithout an explicit request to remove the first virtual annotation(e.g., the electronic device does not detect a user input requestingdeletion of the first virtual annotation). An example of removal ofvirtual annotations in accordance with movement of the camera(s) by morethan a threshold amount is described herein with reference to FIGS.5CL-5CM. Automatically removing a prior virtual annotation (e.g.,measurement) when the field of view for a subsequent virtual annotationshifts by more than a threshold amount reduces clutter in the augmentedreality user interface and avoids potential problems with the fidelityof tracking and displaying prior virtual annotations that are not nearnew virtual annotations. Performing an operation when a set ofconditions has been met without requiring further user input enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, in response to receiving the second set of one ormore inputs associated with the representation of the physical space(1324), in accordance with a determination that the second set of one ormore inputs correspond to a request to shift a field of view of at leastone of the one or more cameras so that the first virtual annotation isno longer visible for more than a threshold amount of time (e.g.,shifting a field of view of at least one of the one or more cameras awayfrom the portion of the physical space to which the first virtualannotation appears to be linked for more than a threshold amount oftime), the electronic device removes the first virtual annotation fromthe representation of the physical space. In some embodiments, the firstvirtual annotation is removed without an explicit request to remove thefirst virtual annotation (e.g., the device does not detect a user inputrequesting deletion of the first virtual annotation). An example ofremoval of virtual annotations in accordance with movement of thecamera(s) so that the virtual annotations are no longer visible for morethan a threshold amount of time is described herein with reference toFIGS. 5CL-5CM. Automatically removing a prior virtual annotation (e.g.,measurement) when the field of view for a subsequent virtual annotationshifts by more than a threshold amount of time reduces clutter in theaugmented reality user interface and avoids potential problems with thefidelity of tracking and displaying prior virtual annotations that arenot near new virtual annotations. Performing an operation when a set ofconditions has been met without requiring further user input enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, while receiving the second set of one or moreinputs and while the first virtual annotation is in the representationof the physical space, the electronic device outputs (1326) anindication that further input will cause the first virtual annotation tobe removed from the representation of the physical space. In someembodiments, the indication includes a graphical indication, an audioindication, and/or a tactile indication. For example, as shown in FIGS.5BW-5BX, device 100 changes the appearance of measurement segment 5212to indicate that further input will cause measurement segment 5212 to beremoved from the representation of physical space 5000 in user interface5006. In another example, as shown in FIGS. 5CA-5CB, device 100 changesthe appearance of measurement segment 5222 to indicate that furtherinput will cause measurement segment 5222 to be removed from therepresentation of physical space 5000 in user interface 5006. Providingvisual, audio, and/or haptic feedback that further input will delete theprior virtual annotation (e.g., measurement) provides an opportunity toalter the input so that the prior virtual annotation is not accidentallyremoved. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the indication is (1328) a visual indication thatis displayed in a predetermined portion of the user interface that isused for displaying alerts (e.g., a designated alert area in the userinterface, such as the area in user interface 5006 above reticle 5010(e.g., the area in which error message 5248 is displayed in FIG. 5CM)).Providing visual feedback in a designated area of the user interface fordisplaying alerts increases the likelihood that a user will see andunderstand the feedback/alert. Providing improved feedback enhances theoperability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, the indication is (1330) a change in appearance ofthe first virtual annotation in the representation of the physical space(e.g., changing a line color, line style, line thickness, fill, opacity,etc.). For example, as shown in FIGS. 5BW-5BX, the appearance (e.g.,line color and line style) of measurement segment 5212 is changed. Insome embodiments, the change in appearance of the first virtualannotation increases in magnitude as a reticle for point placement movesaway from the first virtual annotation). Providing visual feedback bychanging the appearance of the first virtual annotation increases thelikelihood that a user will see and understand that the first virtualannotation will be removed if the same input continues. Providingimproved feedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, the first virtual annotation is (1332) arepresentation of a measurement that includes a description of themeasurement (e.g., a text description of the distance or area measuredby the measurement), and the change in appearance of the first virtualannotation includes removing the description from the representation ofthe measurement while maintaining at least a portion of therepresentation of the measurement. For example, as shown in FIG. 5BV,the label associated with measurement segment 5212 is removed whenreticle 5010 is moved away from measurement segment 5212. Providingvisual feedback by removing a label or other description from ameasurement while maintaining a line or other shape that represents themeasurement increases the likelihood that a user will see and understandthat the measurement will be removed if the same input continues.Providing improved feedback enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, after displaying the change in appearance of thefirst virtual annotation in the representation of the physical space(1334), the electronic device detects an input at a location within athreshold distance (e.g., the same threshold distance that will be usedto determine whether or not to remove the first virtual annotation fromthe display when the second virtual annotation is created) from thefirst virtual annotation (e.g., hovering the reticle for point placementon or near the first virtual annotation), and, in response to detectingthe input at the location within the threshold distance form the firstvirtual annotation, reverses the change in appearance of the firstvirtual indication (e.g., to indicate that if the second virtualannotation includes a portion at the location within the thresholddistance from the first virtual annotation, the first virtual annotationwill not be removed when the second virtual annotation is created). Forexample, the change in appearance of measurement segment 5222 from FIGS.5CA-5CB is reversed in FIG. 5CC when focus point 5012 snaps to a pointon measurement segment 5222. After providing visual feedback to indicatethat the prior virtual annotation will be removed, and after a useralters their input by making the input closer to or on the prior virtualannotation, providing feedback to indicate that the prior virtualannotation will be maintained helps guide the user's input. Providingimproved feedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 13A-13C have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 900, 1000, 1100, 1200, and 1400) are alsoapplicable in an analogous manner to method 1300 described above withrespect to FIGS. 13A-13C. For example, the inputs, user interfaceelements (e.g., measurement points, measurement segments, virtualannotations, representations of the physical space or field of view,affordances, alerts, indicators, labels, anchor points, and/or placementuser interface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 1300optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 900, 1000, 1100, 1200, and 1400). For brevity, thesedetails are not repeated here.

FIGS. 14A-14D are flow diagrams illustrating method 1400 of indicatingwhether objects in a physical space have been identified as objectswhose corresponding representations in an augmented reality environmentcan be tracked in accordance with some embodiments. Method 1400 isperformed at an electronic device (e.g., portable multifunction device100 (FIG. 1A), device 300 (FIG. 3A), or computer system 301 (FIG. 3B))that includes one or more input devices (e.g., touch screen 112 (FIG.1A), touchpad 355 (FIG. 3A), or input device(s) 302 (FIG. 3B), or aphysical button that is separate from the display), one or more displaydevices (e.g., touch screen 112 (FIG. 1A), display 340 (FIG. 3A), ordisplay generation component(s) 304 (FIG. 3B)), and one or more cameras(e.g., optical sensor(s) 164 (FIG. 1A) or camera(s) 305 (FIG. 3B)),optionally one or more sensors to detect intensities of contacts with atouch-sensitive surface of the input device (e.g., contact intensitysensor(s) 165, FIG. 1A), and optionally one or more tactile outputgenerators (e.g., tactile output generator(s) 163 (FIG. 1A) or tactileoutput generator(s) 357 (FIG. 3A)). Some operations in method 1400 are,optionally, combined and/or the order of some operations is, optionally,changed.

As described below, method 1400 provides visual feedback while placingvirtual annotations (e.g., virtual measurements) in an augmented realityenvironment. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

The electronic device displays (1402), via the one or more displaydevices, an annotation placement user interface. The annotationplacement user interface includes (1404): a representation of a physicalspace (e.g., a live preview of a portion of the physical space that isin the field of view of at least one of the one or more cameras); and aplacement user interface element (e.g., a placement indicator) thatindicates a location at which a virtual annotation will be placed in therepresentation of the physical space in response to detecting anannotation placement input (e.g., a tap on a drop point button or a tapon the placement user interface element). For example, as shown in FIG.5CN, device 100 displays user interface 5006 that includes arepresentation of physical space 5000 and a placement user interfaceelement in the form of reticle 5010 in conjunction with focus point 5012that indicates a location at which a virtual annotation will be placedin the representation of physical space 5000 in response to detecting anannotation placement input (e.g., a tap on measurement addition button5014 or, in some embodiments, a tap on reticle 5010 and/or focus point5012).

While displaying the annotation placement user interface, the electronicdevice detects (1406) movement of at least one of the one or morecameras relative to the physical space. The movement of at least one ofthe one or more cameras starts (1408) while the placement user interfaceelement is displayed at a location in the representation of the physicalspace that corresponds to a first portion of the physical space. In someembodiments, the movement includes one or more of moving laterally(e.g., moving up, down, left, right), rotating (e.g., rotating right,left, up, down), or moving forward or backward.

In response to detecting the movement of at least one of the one or morecameras relative to the physical space, the electronic device moves(1410) the placement user interface element to a location in therepresentation of the physical space that corresponds to a secondportion of the physical space that is different from the first portionof the physical space, and updates an appearance of the annotationplacement user interface in accordance with the movement of at least oneof the one or more cameras relative to the physical space, including: inaccordance with a determination that the electronic device is unable toidentify an object in the second portion of the physical space whosecorresponding object in the representation of the physical space can belinked to a virtual annotation, ceasing (1412) to display at least aportion of the placement user interface element; and in accordance witha determination that the device has identified an object in the secondportion of the physical space whose corresponding object in therepresentation of the physical space can be linked to a virtualannotation, maintaining (1414) display of the placement user interfaceelement. For example, as shown in FIG. 5CN, reticle 5010 and focus point5012 are displayed in accordance with a determination that device 100has identified an object (table 5200) in physical space 5000 such thatmeasurements can be added to the representation of table 5200 in userinterface 5006. In FIG. 5CO, reticle 5010 and focus point 5012 are notdisplayed in accordance with a determination that device 100 is unableto identify such an object in physical space 5000. Providing visualfeedback that the electronic device is unable to identify an object inthe physical space whose corresponding object in the representation ofthe physical space can be linked to a virtual annotation (e.g., byceasing to display at least a portion of the placement user interfaceelement) informs a user that the field of view needs to be changed (bymoving the electronic device) until such an object is identified.Providing improved feedback enhances the operability of the device andmakes the user-device interface more efficient (e.g., by helping theuser to provide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, in response to detecting the movement of at leastone of the one or more cameras relative to the physical space (1416): inaccordance with a determination that the placement user interfaceelement is at a location in the representation of the physical spacethat corresponds to a first object in the physical space that is a firstdistance away from one of the one or more cameras, the electronic devicedisplays the placement user interface element at a first size; and inaccordance with a determination that the placement user interfaceelement is at a location in the representation of the physical spacethat corresponds to a second object in the physical world that is asecond distance away from one of the one or more cameras, the electronicdevice displays the placement user interface element at a second size.In some embodiments, the first distance is greater than the seconddistance and the first size is less than the second size. For example,as shown in FIGS. 5BO and 5BT, reticle 5010 and focus point 5012 aredisplayed at smaller respective sizes when positioned over a location inthe live preview that corresponds to a further point in physical space5000 (as shown in FIG. 5BO) than when positioned over a location in thelive preview that corresponds to a closer point in physical space 5000(as shown in FIG. 5BT). In another example, as shown in FIGS. 5CG and5CJ, reticle 5010 and focus point 5012 are displayed at smallerrespective sizes when positioned over a location in the live previewthat corresponds to a further point in physical space 5000 (as shown inFIG. 5CG) than when positioned over a location in the live preview thatcorresponds to a closer point in physical space 5000 (as shown in FIG.5CJ). When the placement user interface element is at the location of agiven object in the live view from a camera, the electronic deviceadjusts the size of the placement user interface element based on thedistance from the camera to the given object in the physical space. Thisvisual feedback provides information to a user about the relativepositions of objects in the physical space. Providing improved feedbackenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, the first size of the placement user interfaceelement in the representation of the physical space is (1418) largerrelative to the first object in the physical space than the second sizeof the placement user interface element in the representation of thephysical space relative to the second object in the physical space(e.g., as described herein with reference to FIG. 5BT). Although in someembodiments the size of the placement user interface element is based onthe distance from the camera to the given object in the physical space,to avoid having the size be too small when an object is far away fromthe camera (or be too large when the object is very close to thecamera), the size in some embodiments does not scale precisely withdistance. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments, the placement user interface element includes(1420) a first portion (e.g., a reticle, such as reticle 5010 in FIG.5CC) and a second portion (e.g., a dot, such as focus point 5012 in FIG.5CC, or other marker). In some embodiments, in response to detecting themovement of at least one of the one or more cameras relative to thephysical space: in accordance with a determination that the placementuser interface element is at a location in the representation of thephysical space that corresponds to a predefined type of feature in thephysical space (e.g., a corner, line or other feature that the secondportion of the placement user interface element can snap to), theelectronic device updates the appearance of the placement user interfaceelement so that the second portion of the placement user interfaceelement is enlarged relative to the first portion of the placement userinterface element; and in accordance with a determination that theplacement user interface element is at a location in the representationof the physical space that does not correspond to a predefined type offeature in the physical space (e.g., a corner, line or other featurethat the second portion of the placement user interface element can snapto), the electronic device maintains display of the placement userinterface element without enlarging the second portion of the placementuser interface element relative to the first portion of the placementuser interface element. For example, as shown in FIGS. 5CC and 5CE, whenfocus point is snapped to an anchor point (such as a point of intereston a previously-added measurement or a point of interest on arepresentation of a physical object), focus point 5012 is enlargedrelative to reticle 5010 (and relative to the size of focus point 5012when not snapped to an anchor point). Changing the appearance of theplacement user interface element (e.g., by increasing the size of aplacement dot relative to the rest of the placement user interfaceelement) when it is over a feature that a virtual annotation point canbe snapped to makes it easy to add a virtual annotation point at thatfeature. Providing improved feedback enhances the operability of thedevice and makes the user-device interface more efficient (e.g., byhelping the user to provide proper inputs and reducing user mistakeswhen operating/interacting with the device).

In some embodiments in which the placement user interface elementincludes (1422) a first portion (e.g., a reticle) and a second portion(e.g., a dot or other marker), in response to detecting the movement ofat least one of the one or more cameras relative to the physical space:in accordance with a determination that the placement user interfaceelement is at a location in the representation of the physical spacethat corresponds to a predefined type of feature in the physical space(e.g., a corner, line or other feature that the second portion of theplacement user interface element can snap to), the electronic deviceupdates the appearance of the placement user interface element so thatthe second portion of the placement user interface element is shiftedrelative to the first portion of the placement user interface element;and in accordance with a determination that the placement user interfaceelement is at a location in the representation of the physical spacethat does not correspond to a predefined type of feature in the physicalspace (e.g., a corner, line or other feature that the second portion ofthe placement user interface element can snap to), the electronic devicemaintains display of the placement user interface element withoutshifting the second portion of the placement user interface elementrelative to the first portion of the placement user interface element.For example, as shown in FIGS. 5CC and 5CE, when focus point is snappedto an anchor point (such as a point of interest on a previously-addedmeasurement or a point of interest on a representation of a physicalobject), focus point 5012 is shifted relative to reticle 5010 (andrelative to the position of focus point 5012 within reticle 5010 whennot snapped to an anchor point). Changing the appearance of theplacement user interface element when it is over a feature that avirtual annotation point can be snapped to (e.g., by shifting thelocation of a placement dot relative to the rest of the placement userinterface element, so that the placement dot snaps to the feature) makesit easy to add a virtual annotation point at that feature. Providingimproved feedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

In some embodiments, in response to detecting the movement of at leastone of the one or more cameras relative to the physical space (1424), inaccordance with a determination that the device is unable to identify anobject in the second portion of the physical space whose correspondingobject in the representation of the physical space can be linked to avirtual annotation, the electronic device displays an alert (e.g.,separate from or overlaid on the annotation placement user interface)with information indicating that the electronic device is unable toidentify an object in the second portion of the physical space whosecorresponding object in the representation of the physical space can belinked to a virtual annotation. For example, as shown in FIG. 5CO, inresponse to movement of device 100 further away from table 5200 device100 displays error message 5250 to indicate that device 100 is unable toidentify an object in physical space 5000 whose corresponding object inthe representation of physical space 5000 in user interface 5006 can belinked to a virtual annotation (e.g., a virtual measurement). Displayingan alert that the electronic device is unable to identify an object inthe physical space whose corresponding object in the representation ofthe physical space can be linked to a virtual annotation informs a userthat the field of view needs to be changed (by moving the electronicdevice) until such an object is identified. Providing improved feedbackenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, in response to detecting the movement of at leastone of the one or more cameras relative to the physical space (1426), inaccordance with a determination that the device is unable to identify anobject in the second portion of the physical space whose correspondingobject in the representation of the physical space can be linked to avirtual annotation, the electronic device displays an alert (e.g.,separate from or overlaid on the placement user interface) withinformation indicating a reason that the electronic device is unable toidentify an object in the second portion of the physical space whosecorresponding object in the representation of the physical space can belinked to a virtual annotation. In some embodiments, the alert includesinformation describing steps that can be taken to improve the ability ofthe electronic device to identify objects in the physical space. Forexample, as shown in FIG. 5CO, in response to movement of device 100further away from table 5200 device 100 displays error message 5250 toindicate that device 100 is unable to identify an object because device100 needs to be moved closer to objects in the field of view of thecamera. Displaying an alert that: (1) explains why the electronic deviceis unable to identify an object in the physical space whosecorresponding object in the representation of the physical space can belinked to a virtual annotation, and/or (2) informs a user how the fieldof view needs to be changed (e.g., by moving the electronic device)helps to correct this situation. Providing improved feedback enhancesthe operability of the device and makes the user-device interface moreefficient (e.g., by helping the user to provide proper inputs andreducing user mistakes when operating/interacting with the device).

In some embodiments, the information indicating the reason that thedevice is unable to identify an object in the second portion of thephysical space whose corresponding object in the representation of thephysical space can be linked to a virtual annotation includes (1428) oneor more of: an indication that more light is required (e.g., as shown inFIG. 5D), an indication that at least one of the one or more cameras (orthe electronic device) is moving too fast, an indication that at leastone of the one or more cameras (or the electronic device) needs to bemoved to locate a surface in the physical space (e.g., as shown in FIG.5E), an indication that at least one of the one or more cameras (or theelectronic device) needs to be moved further away from objects in thephysical space, and an indication that at least one of the one or morecameras (or the electronic device) needs to be moved closer to objectsin the physical space (e.g., as shown in FIG. 5CO).

In some embodiments, while displaying the placement user interfaceelement at a location over the representation of the physical space thatcorresponds to the second portion of the physical space, the electronicdevice detects (1430) a placement input (e.g., a tap gesture on theplacement user interface element or a button that triggers placement ofvirtual annotations in the annotation placement user interface). In someembodiments, in response to detecting the placement input, theelectronic device places at least a portion of a virtual annotation inthe representation of the physical space at a location that correspondsto the placement user interface element. In some embodiments, theplacement of the portion of the annotation includes dropping a firstpoint in a measurement. In some embodiments, the placement of theportion of the annotation includes dropping a second or third point in ameasurement. In some embodiments, the placement of the portion of theannotation includes completing placement of a measurement in therepresentation of the physical space. For example, as shown in FIGS.5BP-5BQ, touch input 5204 (e.g., a tap gesture) on reticle 5010 andfocus point 5012 triggers placement of measurement point 5206 in therepresentation of physical space 5000 in user interface 5006 at thelocation that corresponds to focus point 5012. Placing a virtualannotation point at the displayed location of the placement userinterface element, in response to a placement input, makes it easy toposition the virtual annotation point at the correct location in therepresentation of the physical space. Providing improved visual feedbackenhances the operability of the device and makes the user-deviceinterface more efficient (e.g., by helping the user to provide properinputs and reducing user mistakes when operating/interacting with thedevice).

In some embodiments, while displaying the placement user interfaceelement at a location over the representation of the physical space thatcorresponds to the second portion of the physical space, the electronicdevice detects (1432) an input at a location that corresponds to theplacement user interface element (e.g., for an electronic device with atouch-sensitive display, a tap gesture on the placement user interfaceelement). In some embodiments, in response to detecting the input at thelocation that corresponds to the placement user interface element, theelectronic device displays a graphical indication adjacent to (or near)a different user interface element in the annotation placement userinterface that indicates that activation of the different user interfaceelement (e.g., for a touch-sensitive display, by a tap gesture on thedifferent user interface element) will cause placement of at least aportion of a virtual annotation in the representation of the physicalspace at a location that corresponds to the placement user interfaceelement (e.g., the electronic device displays instructions to tap abutton to drop a point, where the button is different from the placementuser interface element and is located away from the placement userinterface element). For example, as shown in FIG. 5BR, touch input 5204(e.g., a tap gesture) on reticle 5010 and focus point 5012 results indisplay of instruction message 5208 near measurement addition button5014 to indicate that activation of measurement addition button 5014will cause placement of a measurement point in the representation ofphysical space 5000 in user interface 5006 at the location thatcorresponds to focus point 5012. In some embodiments, when a user triesto create a virtual annotation point by tapping on the placement userinterface element (rather than tapping on a different element, such as a“+,” “add,” or similar element), the electronic device displays amessage next to the “+,” “add,” or similar element indicating that thisis the correct element to tap on to create a virtual annotation point(instead of tapping on the placement user interface element). Providingimproved feedback enhances the operability of the device and makes theuser-device interface more efficient (e.g., by helping the user toprovide proper inputs and reducing user mistakes whenoperating/interacting with the device).

It should be understood that the particular order in which theoperations in FIGS. 14A-14D have been described is merely an example andis not intended to indicate that the described order is the only orderin which the operations could be performed. One of ordinary skill in theart would recognize various ways to reorder the operations describedherein. Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 600, 700, 800, 900, 1000, 1100, 1200, and 1300) are alsoapplicable in an analogous manner to method 1400 described above withrespect to FIGS. 14A-14D. For example, the inputs, user interfaceelements (e.g., measurement points, measurement segments, virtualannotations, representations of the physical space or field of view,affordances, alerts, indicators, labels, anchor points, and/or placementuser interface elements such as a reticle and dot), tactile outputs, andintensity thresholds described above with reference to method 1400optionally have one or more of the characteristics of the inputs, userinterface elements, tactile outputs, and intensity thresholds describedherein with reference to other methods described herein (e.g., methods600, 700, 800, 900, 1000, 1100, 1200, and 1300). For brevity, thesedetails are not repeated here.

The operations described above with reference to FIGS. 6A-6C, 7A-7E,8A-8C, 9A-9B, 10A-10B, 11A-11B, 12A-12C, 13A-13C, and 14A-14D are,optionally, implemented by components depicted in FIGS. 1A-1B. Forexample, detecting operations 606, 614, 706, 708, 812, 906, 1006, 1106,1206, 1334, 1430, and 1432, and receiving operations 1304 and 1310 are,optionally, implemented by event sorter 170, event recognizer 180, andevent handler 190. Event monitor 171 in event sorter 170 detects acontact on touch-sensitive display 112, and event dispatcher module 174delivers the event information to application 136-1. A respective eventrecognizer 180 of application 136-1 compares the event information torespective event definitions 186, and determines whether a first contactat a first location on the touch-sensitive display (or whether rotationof the device) corresponds to a predefined event or sub-event, such asselection of an object on a user interface, or rotation of the devicefrom one orientation to another. When a respective predefined event orsub-event is detected, event recognizer 180 activates an event handler190 associated with the detection of the event or sub-event. Eventhandler 190 optionally uses or calls data updater 176 or object updater177 to update the application internal state 192. In some embodiments,event handler 190 accesses a respective GUI updater 178 to update whatis displayed by the application. Similarly, it would be clear to aperson having ordinary skill in the art how other processes can beimplemented based on the components depicted in FIGS. 1A-1B.

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best use the invention and variousdescribed embodiments with various modifications as are suited to theparticular use contemplated.

1. A method, comprising: at an electronic device with a touch-sensitivedisplay and one or more cameras: displaying, on the touch-sensitivedisplay, a user interface of an application, wherein: the user interfaceincludes a representation of a field of view of at least one of the oneor more cameras; the representation of the field of view is displayed ata first magnification; the representation of the field of view isupdated over time based on changes to current visual data detected by atleast one of the one or more cameras; and the field of view includes atleast a portion of a three-dimensional space; while displaying therepresentation of the field of view, detecting a first touch input onthe touch-sensitive display; in response to detecting the first touchinput, adding and displaying on the touch-sensitive display ameasurement point at a first location in the representation of the fieldof view that corresponds to a first location in the three-dimensionalspace; and after adding the measurement point and while continuing todisplay the representation of the field of view: as the field of view ofthe one or more cameras moves, updating the displayed representation ofthe field of view, and continuing to display on the touch-sensitivedisplay the measurement point at a location in the representation of thefield of view that corresponds to the first location in thethree-dimensional space; and detecting a second touch input at alocation on the touch-sensitive display that corresponds to a currentlocation of the measurement point in the representation of the field ofview; in response to detecting the second touch input, enlarging displayof at least a portion of the representation of the field of view fromthe first magnification to a second magnification, greater than thefirst magnification, wherein: the enlarged display of the portion of therepresentation of the field of view includes the current location of themeasurement point in the field of view; and the enlarged display of theportion of the representation of the field of view includes the locationof the second touch input; while displaying the enlarged portion of therepresentation of the field of view, and while continuing to detect thesecond touch input: detecting movement of the field of view of the oneor more cameras or movement of the second touch input; and in responseto detecting movement of the field of view of the one or more cameras ormovement of the second touch input, moving the measurement point withinthe representation of the field of view in accordance with the movementof the field of view of the one or more cameras or the movement of thesecond touch input while continuing to display on the touch-sensitivedisplay an enlarged portion of the representation of the field of viewthat includes the measurement point.
 2. The method of claim 1, wherein:the one or more cameras are located at a first fixed position on theelectronic device on a back side of the electronic device opposite thetouch-sensitive display on a front side of the electronic device, thetouch-sensitive display including a second fixed position that isclosest to the first fixed position on the back side of the device; theuser interface includes: a first affordance that is displayed in aportion of the touch-sensitive display that is away from the secondfixed position and which, when activated, adds a measurement point tothe displayed representation of the field of view, and one or moresecond affordances displayed in accordance with a first orientation ofthe electronic device; and the method includes: detecting movement ofthe electronic device to a second orientation; and in response todetecting the movement of the electronic device to the secondorientation: updating display of the one or more second affordances inaccordance with the second orientation of the electronic device; andcontinuing to display the first affordance in the portion of thetouch-sensitive display that is away from the second fixed position. 3.The method of claim 1, wherein: the user interface of the applicationincludes one or more affordances that correspond to operations of theapplication, including a capture affordance; and the method includes:detecting a third touch input on the capture affordance; and inaccordance with a determination that the third touch input meets firstmedia capture criteria, initiating capture of media of a first type thatincludes a sequence of images of the field of view of at least one ofthe one or more cameras.
 4. The method of claim 3, including: inaccordance with a determination that the third touch input meets secondmedia capture criteria, initiating capture of media of a second typethat includes a single image of the field of view of at least one of theone or more cameras.
 5. The method of claim 1, including: prior todisplaying the user interface of the application: displaying a controlpanel user interface that includes a plurality of controls, wherein afirst control in the plurality of controls corresponds to theapplication; and detecting a touch input activating the first control;and, in response to detecting the touch input activating the firstcontrol, displaying the user interface of the application.
 6. The methodof claim 1, wherein: the user interface is a first user interface of aplurality of user interfaces in the application; the first userinterface corresponds to a measurement mode of the application; and asecond user interface of the plurality of user interfaces corresponds toa levelling mode of the application.
 7. The method of claim 1,including: determining a distance between the electronic device and thefirst location in the three-dimensional space that corresponds to themeasurement point; wherein: in accordance with a determination that thedetermined distance is less than a first threshold distance, amagnification factor between the first magnification and the secondmagnification is a first magnification factor; in accordance with adetermination that the determined distance is greater than a secondthreshold distance, the magnification factor is a second magnificationfactor; and in accordance with a determination that the determineddistance is between the first threshold distance and the secondthreshold distance, the magnification factor is a respectivemagnification factor, between the first magnification factor and thesecond magnification factor, that depends on the determined distance. 8.The method of claim 1, including: while displaying the enlarged displayof at least the portion of the representation of the field of view,detecting a fourth touch input that includes detecting a contact at thecurrent location of the measurement point and detecting movement of thecontact across the touch-sensitive display; and, in response todetecting the movement of the contact across the touch-sensitivedisplay: moving the measurement point across the representation of thefield of view in accordance with the movement of the contact in thefourth touch input.
 9. The method of claim 1, wherein: the first touchinput is detected on an affordance which, when activated, adds ameasurement point to the displayed representation of the field of view;the affordance is displayed in the user interface at a predefinedlocation; and a location in the representation of the field of viewwhere the measurement point is initially added, in response toactivation of the affordance, is distinct from the predefined locationof the affordance.
 10. An electronic device, comprising: atouch-sensitive display; one or more cameras; one or more processors;and memory storing one or more programs, wherein the one or moreprograms are configured to be executed by the one or more processors,the one or more programs including instructions for: displaying, on thetouch-sensitive display, a user interface of an application, wherein:the user interface includes a representation of a field of view of atleast one of the one or more cameras; the representation of the field ofview is displayed at a first magnification; the representation of thefield of view is updated over time based on changes to current visualdata detected by at least one of the one or more cameras; and the fieldof view includes at least a portion of a three-dimensional space; whiledisplaying the representation of the field of view, detecting a firsttouch input on the touch-sensitive display; in response to detecting thefirst touch input, adding and displaying on the touch-sensitive displaya measurement point at a first location in the representation of thefield of view that corresponds to a first location in thethree-dimensional space; and after adding the measurement point andwhile continuing to display the representation of the field of view: asthe field of view of the one or more cameras moves, updating thedisplayed representation of the field of view, and continuing to displayon the touch-sensitive display the measurement point at a location inthe representation of the field of view that corresponds to the firstlocation in the three-dimensional space; and detecting a second touchinput at a location on the touch-sensitive display that corresponds to acurrent location of the measurement point in the representation of thefield of view; in response to detecting the second touch input,enlarging display of at least a portion of the representation of thefield of view from the first magnification to a second magnification,greater than the first magnification, wherein: the enlarged display ofthe portion of the representation of the field of view includes thecurrent location of the measurement point in the field of view; and theenlarged display of the portion of the representation of the field ofview includes the location of the second touch input; while displayingthe enlarged portion of the representation of the field of view, andwhile continuing to detect the second touch input: detecting movement ofthe field of view of the one or more cameras or movement of the secondtouch input; and in response to detecting movement of the field of viewof the one or more cameras or movement of the second touch input, movingthe measurement point within the representation of the field of view inaccordance with the movement of the field of view of the one or morecameras or the movement of the second touch input while continuing todisplay on the touch-sensitive display an enlarged portion of therepresentation of the field of view that includes the measurement point.11. A non-transitory computer readable storage medium storing one ormore programs, the one or more programs comprising instructions which,when executed by an electronic device that includes a touch-sensitivedisplay and one or more cameras, cause the electronic device to:display, on the touch-sensitive display, a user interface of anapplication, wherein: the user interface includes a representation of afield of view of at least one of the one or more cameras; therepresentation of the field of view is displayed at a firstmagnification; the representation of the field of view is updated overtime based on changes to current visual data detected by at least one ofthe one or more cameras; and the field of view includes at least aportion of a three-dimensional space; while displaying therepresentation of the field of view, detect a first touch input on thetouch-sensitive display; in response to detecting the first touch input,add and display on the touch-sensitive display a measurement point at afirst location in the representation of the field of view thatcorresponds to a first location in the three-dimensional space; andafter adding the measurement point and while continuing to display therepresentation of the field of view: as the field of view of the one ormore cameras moves, update the displayed representation of the field ofview, and continue to display on the touch-sensitive display themeasurement point at a location in the representation of the field ofview that corresponds to the first location in the three-dimensionalspace; and detect a second touch input at a location on thetouch-sensitive display that corresponds to a current location of themeasurement point in the representation of the field of view; inresponse to detecting the second touch input, enlarge display of atleast a portion of the representation of the field of view from thefirst magnification to a second magnification, greater than the firstmagnification, wherein: the enlarged display of the portion of therepresentation of the field of view includes the current location of themeasurement point in the field of view; and the enlarged display of theportion of the representation of the field of view includes the locationof the second touch input; while displaying the enlarged portion of therepresentation of the field of view, and while continuing to detect thesecond touch input: detect movement of the field of view of the one ormore cameras or movement of the second touch input; and in response todetecting movement of the field of view of the one or more cameras ormovement of the second touch input, move the measurement point withinthe representation of the field of view in accordance with the movementof the field of view of the one or more cameras or the movement of thesecond touch input while continuing to display on the touch-sensitivedisplay an enlarged portion of the representation of the field of viewthat includes the measurement point.
 12. The non-transitory computerreadable storage medium of claim 11, wherein: the one or more camerasare located at a first fixed position on the electronic device on a backside of the electronic device opposite the touch-sensitive display on afront side of the electronic device, the touch-sensitive displayincluding a second fixed position that is closest to the first fixedposition on the back side of the device; the user interface includes: afirst affordance that is displayed in a portion of the touch-sensitivedisplay that is away from the second fixed position and which, whenactivated, adds a measurement point to the displayed representation ofthe field of view, and one or more second affordances displayed inaccordance with a first orientation of the electronic device; and theone or more programs include instructions which, when executed by theelectronic device, cause the electronic device to: detect movement ofthe electronic device to a second orientation; and in response todetecting the movement of the electronic device to the secondorientation: update display of the one or more second affordances inaccordance with the second orientation of the electronic device; andcontinue to display the first affordance in the portion of thetouch-sensitive display that is away from the second fixed position. 13.The non-transitory computer readable storage medium of claim 11,wherein: the user interface of the application includes one or moreaffordances that correspond to operations of the application, includinga capture affordance; and the one or more programs include instructionswhich, when executed by the electronic device, cause the electronicdevice to: detect a third touch input on the capture affordance; and inaccordance with a determination that the third touch input meets firstmedia capture criteria, initiate capture of media of a first type thatincludes a sequence of images of the field of view of at least one ofthe one or more cameras.
 14. The non-transitory computer readablestorage medium of claim 13, wherein the one or more programs includeinstructions which, when executed by the electronic device, cause theelectronic device to: in accordance with a determination that the thirdtouch input meets second media capture criteria, initiate capture ofmedia of a second type that includes a single image of the field of viewof at least one of the one or more cameras.
 15. The non-transitorycomputer readable storage medium of claim 11, wherein the one or moreprograms include instructions which, when executed by the electronicdevice, cause the electronic device to: prior to displaying the userinterface of the application: display a control panel user interfacethat includes a plurality of controls, wherein a first control in theplurality of controls corresponds to the application; and detect a touchinput activating the first control; and, in response to detecting thetouch input activating the first control, display the user interface ofthe application.
 16. The non-transitory computer readable storage mediumof claim 11, wherein: the user interface is a first user interface of aplurality of user interfaces in the application; the first userinterface corresponds to a measurement mode of the application; and asecond user interface of the plurality of user interfaces corresponds toa levelling mode of the application.
 17. The non-transitory computerreadable storage medium of claim 11, wherein the one or more programsinclude instructions which, when executed by the electronic device,cause the electronic device to: determine a distance between theelectronic device and the first location in the three-dimensional spacethat corresponds to the measurement point; wherein: in accordance with adetermination that the determined distance is less than a firstthreshold distance, a magnification factor between the firstmagnification and the second magnification is a first magnificationfactor; in accordance with a determination that the determined distanceis greater than a second threshold distance, the magnification factor isa second magnification factor; and in accordance with a determinationthat the determined distance is between the first threshold distance andthe second threshold distance, the magnification factor is a respectivemagnification factor, between the first magnification factor and thesecond magnification factor, that depends on the determined distance.18. The non-transitory computer readable storage medium of claim 11,wherein the one or more programs include instructions which, whenexecuted by the electronic device, cause the electronic device to: whiledisplaying the enlarged display of at least the portion of therepresentation of the field of view, detect a fourth touch input thatincludes detecting a contact at the current location of the measurementpoint and detecting movement of the contact across the touch-sensitivedisplay; and, in response to detecting the movement of the contactacross the touch-sensitive display: move the measurement point acrossthe representation of the field of view in accordance with the movementof the contact in the fourth touch input.
 19. The non-transitorycomputer readable storage medium of claim 11, wherein: the first touchinput is detected on an affordance which, when activated, adds ameasurement point to the displayed representation of the field of view;the affordance is displayed in the user interface at a predefinedlocation; and a location in the representation of the field of viewwhere the measurement point is initially added, in response toactivation of the affordance, is distinct from the predefined locationof the affordance.
 20. The electronic device of claim 10, wherein: theone or more cameras are located at a first fixed position on theelectronic device on a back side of the electronic device opposite thetouch-sensitive display on a front side of the electronic device, thetouch-sensitive display including a second fixed position that isclosest to the first fixed position on the back side of the device; theuser interface includes: a first affordance that is displayed in aportion of the touch-sensitive display that is away from the secondfixed position and which, when activated, adds a measurement point tothe displayed representation of the field of view, and one or moresecond affordances displayed in accordance with a first orientation ofthe electronic device; and the one or more programs include instructionsfor: detecting movement of the electronic device to a secondorientation; and in response to detecting the movement of the electronicdevice to the second orientation: updating display of the one or moresecond affordances in accordance with the second orientation of theelectronic device; and continuing to display the first affordance in theportion of the touch-sensitive display that is away from the secondfixed position.
 21. The electronic device of claim 10, wherein: the userinterface of the application includes one or more affordances thatcorrespond to operations of the application, including a captureaffordance; and the one or more programs include instructions for:detecting a third touch input on the capture affordance; and inaccordance with a determination that the third touch input meets firstmedia capture criteria, initiating capture of media of a first type thatincludes a sequence of images of the field of view of at least one ofthe one or more cameras.
 22. The electronic device of claim 21, whereinthe one or more programs include instructions for: in accordance with adetermination that the third touch input meets second media capturecriteria, initiating capture of media of a second type that includes asingle image of the field of view of at least one of the one or morecameras.
 23. The electronic device of claim 10, wherein the one or moreprograms include instructions for: prior to displaying the userinterface of the application: displaying a control panel user interfacethat includes a plurality of controls, wherein a first control in theplurality of controls corresponds to the application; and detecting atouch input activating the first control; and, in response to detectingthe touch input activating the first control, displaying the userinterface of the application.
 24. The electronic device of claim 10,wherein: the user interface is a first user interface of a plurality ofuser interfaces in the application; the first user interface correspondsto a measurement mode of the application; and a second user interface ofthe plurality of user interfaces corresponds to a levelling mode of theapplication.
 25. The electronic device of claim 10, wherein the one ormore programs include instructions for: determining a distance betweenthe electronic device and the first location in the three-dimensionalspace that corresponds to the measurement point; wherein: in accordancewith a determination that the determined distance is less than a firstthreshold distance, a magnification factor between the firstmagnification and the second magnification is a first magnificationfactor; in accordance with a determination that the determined distanceis greater than a second threshold distance, the magnification factor isa second magnification factor; and in accordance with a determinationthat the determined distance is between the first threshold distance andthe second threshold distance, the magnification factor is a respectivemagnification factor, between the first magnification factor and thesecond magnification factor, that depends on the determined distance.26. The electronic device of claim 10, wherein the one or more programsinclude instructions for: while displaying the enlarged display of atleast the portion of the representation of the field of view, detectinga fourth touch input that includes detecting a contact at the currentlocation of the measurement point and detecting movement of the contactacross the touch-sensitive display; and, in response to detecting themovement of the contact across the touch-sensitive display: moving themeasurement point across the representation of the field of view inaccordance with the movement of the contact in the fourth touch input.27. The electronic device of claim 10, wherein: the first touch input isdetected on an affordance which, when activated, adds a measurementpoint to the displayed representation of the field of view; theaffordance is displayed in the user interface at a predefined location;and a location in the representation of the field of view where themeasurement point is initially added, in response to activation of theaffordance, is distinct from the predefined location of the affordance.